Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parent_id attribute for comments from pushshift API #19

Closed
voussoir opened this issue Feb 23, 2023 · 0 comments
Closed

parent_id attribute for comments from pushshift API #19

voussoir opened this issue Feb 23, 2023 · 0 comments

Comments

@voussoir
Copy link
Owner

When requesting comments from the Pushshift API, parent_id is coming back as null for root comments, and integers for reply comments.

Another user reported this issue on reddit: https://www.reddit.com/r/pushshift/comments/ujwdyt/parent_id_is_being_returned_as_integer_bug/

For timesearch, we'll solve it by transforming this data in the dummy object so that by the time it reaches TSDB, it will be normal.

     def __init__(self, **attributes):
         for (key, val) in attributes.items():
             if key == 'author':
                  val = DummyObject(name=val)
             elif key == 'subreddit':
                  val = DummyObject(display_name=val)
             elif key in ['body', 'selftext']:
                  val = html.unescape(val)
+            elif key == 'parent_id':
+                 if val is None:
+                    val = attributes['link_id']
+                 elif isinstance(val, int):
+                    val = 't1_' + common.b36(val)

If you have a timesearch database and you need to repair the parent_id data, you do not need to re-download any of those comments. We would just take all rows with null and copy the submission ID, and all rows with int and run them through the b36 function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant