Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add new harvester service and event worker #68

Merged
merged 3 commits into from
Mar 22, 2022
Merged

add new harvester service and event worker #68

merged 3 commits into from
Mar 22, 2022

Conversation

lyqht
Copy link
Contributor

@lyqht lyqht commented Mar 22, 2022

What this PR does:

  • Address Meetup requires migration from REST to GraphQL #67
  • Feature
    • New harvester service that scrapes the Meetup Website for tech groups in Singapore, parses their respective RSS and format the events to fit the previous harvester worker functionality to insert events into DB. (Original MVP work with more documentation can be found [here])
  • Notes
    • This implementation does not touch the existing GraphQL resolvers
    • This implementation does not populate the Groups table, because the main problem now is to get telegram events notifications working again first.
    • This implementation does not consider blacklist and whitelist groups/events.
    • Previous harvester service & harvester tasks are moved to their respective /deprecated folder for now, I'll leave it to the maintainers to decide what to do with them
    • No tests have been written yet for scraping and rss parsing, but at the same time I don't think it's very necessary.
    • There is a strange console error after running the telegram worker tests, and this is due to the fact that push() is included at the end of the file. I modified this by throwing it to exports and modified the package.json script for telegrambot.

Functionality is checked by running the below command and checking the DB (refer to screenshot)

npm run harvest:events:worker | tee harvest_events_worker.out   

image

@miccheng
Copy link
Member

LGTM. We try first bah.

@miccheng miccheng merged commit 57def03 into engineersftw:main Mar 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants