Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Queuing; Refactor Extractors to Tasks; refactor MemoryStore interface #246

Merged
merged 11 commits into from
Oct 28, 2023

Conversation

danielchalef
Copy link
Member

@danielchalef danielchalef commented Oct 28, 2023

  • Add SQL TaskQueue using Watermill
  • Refactor Extractors to new Task queue model
  • Refactor MemoryStore Interfaces as precursor to refactoring MemoryStore
  • Code cleanup

@danielchalef danielchalef temporarily deployed to build-test October 28, 2023 18:01 — with GitHub Actions Inactive
@danielchalef danielchalef merged commit 53c4822 into main Oct 28, 2023
5 checks passed
@danielchalef danielchalef deleted the feat/queueing branch October 28, 2023 18:09
@moltar
Copy link

moltar commented Nov 5, 2023

@danielchalef wow, looks like a ton of work went into this one. I am not familiar with Go yet to grok all the changes. But does this require any migrations? Or does Go code migrate implicitly when you start the service? Are there any docs for this change? Thanks!!

@danielchalef
Copy link
Member Author

danielchalef commented Nov 5, 2023

@moltar No migrations were necessary. The functional router "Task" schemas remained the same as the prior "Extractor" tasks. The message queue framework I used automatically generates schemas for each handler. Since this change was an internal shift from an observer model using Go channels as queues, to a SQL-based message queue architecture, I've not prioritized documenting it! I will get around to doing so sometime soon!

@moltar
Copy link

moltar commented Nov 5, 2023

Alright thank you for explaining. I was only looking for any end user docs, but sounds like none needed.

I was thinking in the context of applying database migrations manually for this change. But given that it's under the hood implementation detail then is all good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants