Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discord: fetch backwards and add a beginningDate config #3329

Merged
merged 9 commits into from
Mar 2, 2022

Conversation

blueridger
Copy link
Member

Description

Discord's rate limiting is getting harsher, and loading the full history is becoming too long even for smaller communities. This PR attempts to address this by allowing users to set a beginningDate, and then fetching backwards until that date is reached and then stopping. For example, users could choose to only load messages after 1/1/2022.

Test Plan

  1. Add console.log(new Date(messages[0]?.timestamp)) on line 180 of the fetcher.
  2. Optionally comment out lines 117-126 to disable reaction fetching for quicker testing.
  3. yarn build
  4. checkout sourcecred/cred/gh-pages
  5. scdev load and verify that only recent messages are pulled (the cache is respected and not re-pulled)
  6. yarn clean
  7. scdev load and verify messages are pulled in reverse order
  8. ctrl + C to interrupt after a channel has loaded 3 pages
  9. scdev load and verify that a recent page or two is reloaded, but then it skips most of what is in the cache to resume loading before what is in the cache.
  10. yarn clean
  11. add "beginningDate": "1/22/2022" in discord/config.js
  12. scdev load and verify that messages are not loaded once the beginning date is reached

@blueridger blueridger merged commit 67af254 into main Mar 2, 2022
@blueridger blueridger deleted the discord-fetch branch March 2, 2022 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants