Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

the scraper skip some tweets when using --until #12

Closed
minamotorin opened this issue Jul 19, 2022 · 4 comments
Closed

the scraper skip some tweets when using --until #12

minamotorin opened this issue Jul 19, 2022 · 4 comments
Labels
enhancement New feature or request

Comments

@minamotorin
Copy link
Owner

I found that this part of the code in url.py

    if "win" in platform:
        return f'\"{date.split()[0]}\"'

sometimes makes the scraper skip some tweets when using --until "%y-%m-%d %H:%M:%S" on Windows. It starts from some hours before the specified one. Removing these lines seems to achieve better results.

Originally posted by @Tortar in #8 (comment)

@minamotorin
Copy link
Owner Author

@Tortar
The code you pointed out was originally added as a workaround for twintproject#597.
When I fixed twintproject#1136, I changed the one (188e521).
But I doesn't remove the code because I can't deny the possibility that twintproject#597 has recur.

If someone helps debugging on Windows and twintproject#597 resolves, I would be able to remove the code.

@minamotorin minamotorin added the enhancement New feature or request label Jul 19, 2022
@Tortar
Copy link

Tortar commented Jul 19, 2022

I will work on it, for now --until seems to work fine while --since has some problem

@Tortar
Copy link

Tortar commented Jul 20, 2022

Actually...on Windows works perfectly without

    if "win" in platform:
        return f'\"{date.split()[0]}\"'

I tried

twint -s pineapple --until "2020-11-09 9:00:00" --since "2020-11-09 1:00:00"

and

twint -s pineapple --until "2020-11-09" --since "2020-11-08"

and they work as expected, stopping at the right point. So, I think deleting those lines could be an improvement.

minamotorin added a commit that referenced this issue Jul 20, 2022
Remove workaround of twintproject#597
@minamotorin
Copy link
Owner Author

minamotorin commented Jul 20, 2022

I removed the code. 0b07327

@Tortar
Thanks for your help!

minamotorin added a commit that referenced this issue Jul 20, 2022
@minamotorin minamotorin pinned this issue Jul 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants