Fixes #1252: Add Scraper for questions of QuoraScraper #1255

vibhcool · 2017-06-16T20:49:44Z

Fixes Issue: #1252

Test link : http://35.184.148.24/api/quoraprofilescraper?query=Vibhor-Verma-5
http://35.184.148.24/api/quoraprofilescraper?query=asd

I have:

There is a corresponding issue for this pull request.
Mentioned the Issue number in the pull request commit message Fixes #<number> commit message
There is only strictly only one commit per issue.

For the reviewers

I have:

Reviewed this pull request by an authorized contributor.
The reviewer is assigned to the pull request.

vibhcool · 2017-06-20T10:42:31Z

@Orbiter @sudheesh001 @daminisatya @jig08 @mariobehling @singhpratyush @kavithaenair @SKrPl @hemantjadon @Achint08 @sarishinohara @djmgit

I am stuck at a wierd problem, In this method getData(), scraped data is returned to here https://github.com/loklak/loklak_server/pull/1255/files#diff-37e4b654b188b27f598f203ceb234782R83 , after this method is returns it's control.
this method returns no data. but it collects the data.
and json output is generated before scraped data is collected
what shall I do?
EDIT: fixed this, open for review

codecov-io · 2017-07-02T23:25:18Z

Codecov Report

Merging #1255 into development will decrease coverage by 0.09%.
The diff coverage is 2.25%.

@@               Coverage Diff               @@
##             development   #1255     +/-   ##
===============================================
- Coverage           9.08%   8.99%   -0.1%     
+ Complexity           397     396      -1     
===============================================
  Files                200     200             
  Lines              17359   17506    +147     
  Branches            3249    3267     +18     
===============================================
- Hits                1577    1574      -3     
- Misses             15475   15623    +148     
- Partials             307     309      +2

Impacted Files	Coverage Δ	Complexity Δ
src/org/loklak/objects/Timeline2.java	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
src/org/loklak/api/search/QuoraProfileScraper.java	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
src/org/loklak/harvester/BaseScraper.java	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
src/org/loklak/api/search/ConsoleService.java	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
src/org/loklak/harvester/Post.java	`30.55% <18.18%> (-22.08%)`	`2 <1> (ø)`
src/org/json/JSONTokener.java	`42.75% <0%> (-1.38%)`	`22% <0%> (-1%)`
src/org/loklak/harvester/TwitterScraper.java	`52.73% <0%> (-0.48%)`	`35% <0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 12d7cda...39980af. Read the comment docs.

daminisatya

Again the same issue. You are partially providing fixes to two or three issues together and creating multiple PRs. If u think each fix is different then it's the problem with you on dividing a huge chunk of problem into smaller issues.

vibhcool · 2017-07-04T09:31:38Z

@daminisatya please see PR description, :)

vibhcool · 2017-07-04T11:33:53Z

@Achint08 @djmgit @hemantjadon @kavithaenair @SKrPl @singhpratyush @daminisatya : please review, I have rebased the PR

singhpratyush

Left a few comments. Please take a look.

Also, please update the deployment link.

singhpratyush · 2017-07-04T14:01:03Z

src/org/loklak/api/search/QuoraProfileScraper.java

@@ -37,12 +40,12 @@

 public class QuoraProfileScraper extends BaseScraper {

-    private final long serialVersionUID = -3398701925784347310L;
+    private final long serialVersionUID = -3398701925784347312L;


Why do we need to change the serialVersionUID?

I have done this to show that this class is not backward compatible.
https://stackoverflow.com/questions/285793/what-is-a-serialversionuid-and-why-should-i-use-it

singhpratyush · 2017-07-04T14:03:24Z

src/org/loklak/api/search/QuoraProfileScraper.java

+                dataThreads[i].join();
+            }
+        } catch(InterruptedException e) {
+            DAO.severe("Couldn't complete all threads");


Maybe we need to mention the scraper name and type variable here so that it is easy to debug.

singhpratyush · 2017-07-04T14:07:34Z

src/org/loklak/objects/Timeline2.java

@@ -266,14 +266,26 @@ private Timeline2 addPost(Post post) {
        return this;
    }

+    public void mergePost(Timeline2 list) {


Why not using Timeline2#putAll here?

I skipped this method as still UserEntry is not in use. I will have to set up Post object for this.

vibhcool · 2017-07-04T17:18:56Z

@singhpratyush , updated the test link, please see :)

singhpratyush

The search results that I see for http://35.187.215.147/api/quoraprofilescraper?query=ronaldo looks like this -

{
    "metadata": {
        "count": "11",
        "hits": 11
    },
    "peer_hash": "SfjbDL/f/9SH1ySjUSX8k+lZ2ocidIqVgxRpbwv1EA0=",
    "peer_hash_algorithm": "SHA-256",
    "session": {
        "identity": {
            "anonymous": true,
            "name": "10.12.1.1",
            "type": "host"
        }
    },
    "statuses": [
        {
            "bio": "",
            "feeds": {},
            "knows_about": [
                "Ronaldo",
                "Ronaldo",
                "Ronaldo",
                "Ronaldo",
                "Ronaldo"
            ],
            "profileImage": "https://qph.ec.quoracdn.net/main-thumb-86854754-50-apcitmmrwrdgxmcxzspksmqprxzawqzj.jpeg",
            "rss_feed_link": "https://www.quora.com/profile/ronaldo/rss",
            "search_url": "https://www.quora.com/profile/ronaldo",
            "timestamp": 1499189682253,
            "user": "Phillip Michael Mpalabule"
        },
        {
            "post_ques": "What are some famous gestures of respect in sports?",
            "post_type": "question",
            "post_url": "https://www.quora.com//What-are-some-famous-gestures-of-respect-in-sports",
            "search_url": "https://www.quora.com/search/?q=ronaldo&type=question",
            "timestamp": 1499189682080
        },
    ...
}

The users and questions are returned at the same level in JSON, which is not good as they are completely different entities. Somthing like this would be better -

{
    "metadata": {
        ...
        "count": {
            "user": 1,
            "question": 11
        },
        ...
    },
    "users": [
        ...
    ],
    "questions": [
        ...
    ]
}

hemantjadon · 2017-07-05T03:40:07Z

@vibhcool I agree with @singhpratyush 's idea, it will be great if the users and questions are at different level

vibhcool · 2017-07-05T09:15:49Z

@hemantjadon @singhpratyush , instead of arranging different posts according to different types, we can mention the post_type field and scraper_name field.
Though for QuoraScraper, @singhpratyush 's idea is nice. But this will not work when there are numerous scrapers working concurrently. what do you say?

{
    "metadata": {
        ...
        "count": {
            "user": 1,
            "question": 11
        },
        ...
    },
    "posts": [
        {
             ...
             "post_type"="question",
             "scraper_name"="quora"
        },
        {
                "post_type"="profile",
             "scraper_name"="quora"
        },
{
                "post_type"="some other type",
             "scraper_name"="any other scraper"
        }

    ],
    
}

singhpratyush · 2017-07-05T09:24:18Z

@vibhcool: This is not a good practice. If the entities are different, we should not put them in same JSON level.

If we later happen to have lots of services to give as an answer, we can put them under different levels as follows -

{
    "metadata": {
        ...
    },
    "results": {
        "quora": {
            "users": [
                ...
            ],
            "questions": [
                ...
            ]
        },
        "twitter": {
            "statuses": [
                ...
            ]
        },
        ...
    }
}

singhpratyush

See previous comment.

hemantjadon

@vibhcool I agree here with @singhpratyush they should be in different levels

vibhcool · 2017-07-07T10:39:32Z

@daminisatya @singhpratyush @kavithaenair @SKrPl @hemantjadon @Achint08 @djmgit
made the suggested changes , please review

Changes:

Now post object can also be used as wrapper object instead of being a interface.

not changed:

I have not implemented metadata showing posts data as it would make this PR bigger and a bit more complicated to review. I would fix it in subsequent PRs

hemantjadon

LGTM 👍

SKrPl

The topics in knows_about differ in what the actual user knows, for example:
The query was "Android", the user provided is Marc Bodnick:

Things he actually knows about differ from Android (operating system), related screen shot:

Also, the search_url param has value https://www.quora.com/profile/Android which redirects to https://www.quora.com/topic/Android-operating-system, so is it possible to get the final URL (after redirection)?

vibhcool · 2017-07-08T11:59:03Z

@SKrPl yes, I am creating issue for this, these aren't related to PR task :)

…1255)

vibhcool force-pushed the 1230 branch 2 times, most recently from cf66e91 to 17ca182 Compare June 20, 2017 09:01

vibhcool mentioned this pull request Jun 20, 2017

Add Post and BaseScraper to QuoraProfileScraper #1250

Merged

5 tasks

vibhcool force-pushed the 1230 branch from 17ca182 to d936ee8 Compare June 20, 2017 10:04

vibhcool force-pushed the 1230 branch from d936ee8 to 15104eb Compare June 24, 2017 22:38

vibhcool changed the title ~~[WIP] Fixes #1252: Add Scraper for questions of QuoraScraper~~ Fixes #1252: Add Scraper for questions of QuoraScraper Jun 24, 2017

vibhcool force-pushed the 1230 branch 2 times, most recently from b1ec82a to c8b0b58 Compare July 1, 2017 12:57

vibhcool force-pushed the 1230 branch from c8b0b58 to 2756385 Compare July 2, 2017 23:17

vibhcool force-pushed the 1230 branch from 2756385 to a81533b Compare July 2, 2017 23:34

vibhcool mentioned this pull request Jul 3, 2017

Fixes #1288: Add data structure to BaseScraper to access get-parameters #1289

Closed

5 tasks

vibhcool force-pushed the 1230 branch 2 times, most recently from 7562434 to 3bb0261 Compare July 3, 2017 22:06

daminisatya suggested changes Jul 4, 2017

View reviewed changes

vibhcool force-pushed the 1230 branch 4 times, most recently from 083c479 to 3d4d4cf Compare July 4, 2017 10:55

singhpratyush reviewed Jul 4, 2017

View reviewed changes

vibhcool force-pushed the 1230 branch 3 times, most recently from 9749393 to b9f0811 Compare July 4, 2017 14:41

singhpratyush suggested changes Jul 4, 2017

View reviewed changes

mariobehling requested review from sudheesh001, singhpratyush, hemantjadon, SKrPl, Achint08 and kavithaenair July 5, 2017 09:27

singhpratyush suggested changes Jul 5, 2017

View reviewed changes

hemantjadon suggested changes Jul 6, 2017

View reviewed changes

vibhcool force-pushed the 1230 branch 2 times, most recently from e9c0a15 to 85561f8 Compare July 7, 2017 10:26

vibhcool force-pushed the 1230 branch from 85561f8 to 777dae8 Compare July 7, 2017 10:56

Fixes loklak#1252: Add Scraper for questions of QuoraScraper

39980af

vibhcool force-pushed the 1230 branch from 777dae8 to 39980af Compare July 7, 2017 11:04

hemantjadon approved these changes Jul 7, 2017

View reviewed changes

singhpratyush approved these changes Jul 7, 2017

View reviewed changes

kavithaenair approved these changes Jul 7, 2017

View reviewed changes

djmgit approved these changes Jul 7, 2017

View reviewed changes

SKrPl reviewed Jul 8, 2017

View reviewed changes

mariobehling merged commit 4a213c5 into loklak:development Jul 8, 2017

vibhcool deleted the 1230 branch July 8, 2017 12:00

vibhcool added a commit to vibhcool/loklak_server that referenced this pull request Jul 15, 2017

Fixes loklak#1252: Add Scraper for questions of QuoraScraper (loklak#…

2578c1b

…1255)

vibhcool added a commit to vibhcool/loklak_server that referenced this pull request Jul 19, 2017

Fixes loklak#1252: Add Scraper for questions of QuoraScraper (loklak#…

bad0ba6

…1255)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #1252: Add Scraper for questions of QuoraScraper #1255

Fixes #1252: Add Scraper for questions of QuoraScraper #1255

vibhcool commented Jun 16, 2017 •

edited

vibhcool commented Jun 20, 2017 •

edited

codecov-io commented Jul 2, 2017 •

edited

daminisatya left a comment

vibhcool commented Jul 4, 2017

vibhcool commented Jul 4, 2017

singhpratyush left a comment

singhpratyush Jul 4, 2017

vibhcool Jul 4, 2017

singhpratyush Jul 4, 2017

singhpratyush Jul 4, 2017

vibhcool Jul 4, 2017 •

edited

vibhcool commented Jul 4, 2017

singhpratyush left a comment •

edited

hemantjadon commented Jul 5, 2017

vibhcool commented Jul 5, 2017 •

edited

singhpratyush commented Jul 5, 2017

singhpratyush left a comment

hemantjadon left a comment

vibhcool commented Jul 7, 2017 •

edited

hemantjadon left a comment

SKrPl left a comment

vibhcool commented Jul 8, 2017

Fixes #1252: Add Scraper for questions of QuoraScraper #1255

Fixes #1252: Add Scraper for questions of QuoraScraper #1255

Conversation

vibhcool commented Jun 16, 2017 • edited

For the reviewers

vibhcool commented Jun 20, 2017 • edited

codecov-io commented Jul 2, 2017 • edited

Codecov Report

daminisatya left a comment

Choose a reason for hiding this comment

vibhcool commented Jul 4, 2017

vibhcool commented Jul 4, 2017

singhpratyush left a comment

Choose a reason for hiding this comment

singhpratyush Jul 4, 2017

Choose a reason for hiding this comment

vibhcool Jul 4, 2017

Choose a reason for hiding this comment

singhpratyush Jul 4, 2017

Choose a reason for hiding this comment

singhpratyush Jul 4, 2017

Choose a reason for hiding this comment

vibhcool Jul 4, 2017 • edited

Choose a reason for hiding this comment

vibhcool commented Jul 4, 2017

singhpratyush left a comment • edited

Choose a reason for hiding this comment

hemantjadon commented Jul 5, 2017

vibhcool commented Jul 5, 2017 • edited

singhpratyush commented Jul 5, 2017

singhpratyush left a comment

Choose a reason for hiding this comment

hemantjadon left a comment

Choose a reason for hiding this comment

vibhcool commented Jul 7, 2017 • edited

hemantjadon left a comment

Choose a reason for hiding this comment

SKrPl left a comment

Choose a reason for hiding this comment

vibhcool commented Jul 8, 2017

vibhcool commented Jun 16, 2017 •

edited

vibhcool commented Jun 20, 2017 •

edited

codecov-io commented Jul 2, 2017 •

edited

vibhcool Jul 4, 2017 •

edited

singhpratyush left a comment •

edited

vibhcool commented Jul 5, 2017 •

edited

vibhcool commented Jul 7, 2017 •

edited