Skip to content

Enable new queueworker #3786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 76 commits into from
May 13, 2022
Merged

Enable new queueworker #3786

merged 76 commits into from
May 13, 2022

Conversation

pbiggar
Copy link
Member

@pbiggar pbiggar commented May 9, 2022

This started as an attempt to enable the new queueworker, and morphed into replacing the queues entirely.

The old queue was (well, still is) quite janky. It's built on postgres and currently eats about 50% of our database's CPU capacity. It has long been a desire to replace it with Real Queues™️. As I started testing out the transition, it became clear that a direct port of the old queueworkers to the new one was untenable, as we used a single transaction around the entire job. That had been causing problems before, but we now do exceptions on timeouts and have connection pools and multi-threading and such, so as well as seeing actual problems, it seemed like there were enough unknown-unknowns that I wasn't confident switching over.

Hence, made a new plan, with a new design.

The remaining work is documented here. The next step is to manually deploy it to the QueueWorker and see what happens. If that works, we'll merge and get it deployed. Since the pubsub queues are empty at the moment until we enable it on a per-canvas basis, we can check configuration, metrics, etc, while slowly sending some events to it.

LaunchDarkly changes

I also changed how LaunchDarkly works, as it was a little error-prone. Now we add functions to the LibService.LD module, just like we do with LibService.Config. The definition has name, defaults, and test defaults, and has a type ensuring we get the right kind of "user", so it's much hard to fuck up.

Serialization changes

The format we used for storing values in the queue was bad and unsafe, so I made a new one. The new one is a direct serialization of a type, and the type is now separated from whatever RT.Dval happens to be at the time. The old format is still used in other places, but this removes one usage at least.

Misc changes:

  • some CODING-GUIDE additions
  • adding the pubsub emulator to the dockerfile
  • handle some edge cases in Telemetry when spans aren't set yet

@pbiggar pbiggar force-pushed the paul/enable-new-queueworker branch from 3e1f7b4 to 6a38259 Compare May 12, 2022 03:55
@@ -16,7 +16,8 @@
<Compile Include="VendoredTablecloth.fs" />
<Compile Include="RuntimeTypes.fs" />
<Compile Include="RuntimeTypesAst.fs" />
<Compile Include="DvalReprInternal.fs" />
<Compile Include="DvalReprInternalDeprecated.fs" />
<Compile Include="DvalReprInternalNew.fs" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nitpick) maybe without "New"? I'm not sure what value the suffix brings. Assumedly we're going to rip out the Deprecated one before long, at which point we'd strip the "new" anyway? Not a big deal either way.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, could remove that. It was initially to make fully sure that they had all been renamed (eg, get rid of the old name so that the compiler will tell you if you forgot to update something). but now that that's done, there probably isn't a value to the new name.

Actually, that's only true once this merges. Maybe I'll remove this on the next iteration.

@pbiggar pbiggar merged commit 45cd6e7 into main May 13, 2022
@pbiggar pbiggar deleted the paul/enable-new-queueworker branch May 13, 2022 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants