[Scrapper-Service]add g2r scrapper, modify config.json

[ikoala21]

Implements #2

[rajatkb]

Looks great a few remarks from my end

1. Use the logger against inner errors also.

• Check for None errors against accessing None variable , log and raise exception against them
• In that way if the scrapper starts failing we will have a detailed trace of what's going wrong.
• I see you have used the error log in the top level function. Do so in the inner functions as well.

2. you are putting wrong information in the metadata. Metadata is what belongs to the scrapper. You have to follow the Metadata datamodel. The extra information all goes with the Conference kwargs. Look into the Conference datamodel you will find what extra info you should be putting if you find them while scrapping. You can also put your own extra info.

3. Also are you going through the top conferences only. You can go through both the regular one and the top conference one.

[rajatkb]

@sagar-sehgal please put your review for the code.

[thesagarsehgal]

I wasn't able to run the branch since I got the following error.

2020-03-19 01:37:25,742 - plugins.guide2research - ERROR - Error while parsing page http://www.guide2research.com/?p=12801, find full trace local variable 'content' referenced before assignment

Received the same error for all the links.

I have commented on some of the locations due to which I feel the error might have been caused.

[rajatkb]

• Use debug logging. It's for the purpose of addressing future issues , log where ever needed for debugging issues down the line.
  few examples

1. page request succefull log
2. page succefully parsed log
3. data commited in db debug log etc

[rajatkb]

• Use of yield. Though it's not a major issue. But for a large number of links. The yield would help not create large list in memory. You can convert any where there is a list being returned to yield.

• Approving this from my end but incorporate the yield change.

[rajatkb]

• Looks fine from my end. Further if any issues come into play , will raise bug issue for same. Do address them 👍 . Otherwise Great job 👍 😁 , awaiting a go by @sagar-sehgal

[rajatkb]

• Use the provided function, this will block the thread since no timeout is given. Use the given function

[rajatkb]

• Make same change , anywhere you may have missed it. Also post a mongodb schema layout of conference captured by your scrapper , one from top conference , and one from general. Should give us the idea in case any change required.

[ikoala21]

Hey, there's a catch with the get_pagemethod provided, it works fine when the page/route is returned. In case of individual conferences listed on both top conferences and all conferences there's a redirect that's happening and it is not allowed by the provided method. In such a case, using get_page would not work as I wanted it to.

[ikoala21]

I can change the request method to include timeout if that's okay

[thesagarsehgal]

While running the code I came am coming across the following error.

[dill routine]:  Runnable failed in runtime due to error raised HTTPConnectionPool(host='www.guide2research.com', port=80): Read timed out. (read timeout=1)

I will come across this when it initially fetches the pages.

Try running the code after deleting the db.

[rajatkb]

While running the code I came am coming across the following error.

[dill routine]:  Runnable failed in runtime due to error raised HTTPConnectionPool(host='www.guide2research.com', port=80): Read timed out. (read timeout=1)

I will come across this when it initially fetches the pages.

Try running the code after deleting the db.

This should not happen unless

1. The error was not handled
2. The already given method for page request was not provided.

Both of these can lead to this. @manasakalaga address this.

[ikoala21]

The already given method for page request was not provided.

@rajatkb what do you mean by this?

[ikoala21]

Try running the code after deleting the db.

@sagar-sehgal do you mean entries in the db?

EDIT
I've figured out what's happening. the thing is that as I mentioned above, each of the conference pages redirects to it's ultimate destination and the time that's being taken is more than usual. I was able to fix this by adding timeout parameter to my conference page request method

[rajatkb]

• Use yield here , you may risk getting huge amount links in memory

[ikoala21]

Can you suggest a way of going about it by looking at the flow?
Here's an outline

1. First, I'm looking for all pages of all conferences and saving them to all_conf_page_links
2. Then I'm further parsing each of the item in that list to get links individual conferences and adding them to a list.

[rajatkb]

Follow the below pattern

 def some_function_that_return_list( args ):
            while True:
                 ## do some processing 
                 yield result

 def other_func_that_uses_above( args ):
       for result in some_function_that_return_list(new_args):
             ## do something with result
             new_result = process(result)
             yield result

 for result in other_funct_that_uses_above(args):
      print(result)

Where you see your self iterating over a one-timer list, you can resort to yield. like self.all_conf_page_links , can be a yielded instead of a long list. Similarly the processing done on it can be yield also ........ The idea is dont accumalate into a list just to return it. Yield the list..

[rajatkb]

The already given method for page request was not provided.

@rajatkb what do you mean by this?

self.get_page() method ... already has an adaptive timeout routine.
Also you need to handle such timeout. Timeouts will be there and it cannot be avoided. So those need to be put as an exception which must be handled and then move to the next page , or next link.

[ikoala21]

The already given method for page request was not provided.

@rajatkb what do you mean by this?

self.get_page() method ... already has an adaptive timeout routine.
Also you need to handle such timeout. Timeouts will be there and it cannot be avoided. So those need to be put as an exception which must be handled and then move to the next page , or next link.

I get this.
To reach each of the conference links, I need to allow for redirect to happen, which is not provided in the self.get_page

[rajatkb]

To reach each of the conference pages ? Are you reaching out to the actual conference url or the one's provided by guide2research. If it's by guide2research does it still requires redirect ?

[ikoala21]

To reach each of the conference pages ? Are you reaching out to the actual conference url or the one's provided by guide2research. If it's by guide2research does it still requires redirect ?

I'm reaching out to the link provided by Guide 2 Research itself. and, Yes it requires a redirect. Look at the bottom left of the image below

---

Upon clicking it takes me to the following page via a redirect

[rajatkb]

Well that should not require a redirect does it. U can simply request this jew page. That you require. Though adding redirect true is not a big deal . We can simply add that in the Adaptive Request class in utility package. But still would like to know why is it required ? Because if u request the the link given in screenshot it should get download the html content of it. Redirect would required when this link would redirect to a new page (something of tinyurl sort)

PS: Yes I visited the site, I get what you are saying. They are not following rest and are bringing in pages from that body arguments. U can make the change for that redirect in AdaptiveRequest package, in utility class. It's using the request package internally.

[ikoala21]

Any resources on how do I go about adding that adaptive request behavior to the package?

[rajatkb]

Any resources on how do I go about adding that adaptive request behavior to the package?

Just visit that class. It's using the requests package , u can add the same argument or redirect ther also,

[rajatkb]

@manasakalaga, since the ur other bug fix is approved, u can now finish up this one.

[ikoala21]

mongo object for:

1. Top conferences:
   

2. All conferences:
   

[rajatkb]

@manasakalaga looks clean from my end , merging it.

[rajatkb]

• Great JOB ! 👍