Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixing redis queue for tests by adding prefix #921

Merged
merged 6 commits into from
Nov 18, 2021
Merged

Fixing redis queue for tests by adding prefix #921

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
01c3075
fixing redis queue for tests by adding prefix
mboudreau Nov 17, 2021
32ff62d
changing queue prefix for redis prefix so that it fixes it for all re…
mboudreau Nov 17, 2021
3e29be2
Revert "changing queue prefix for redis prefix so that it fixes it fo…
mboudreau Nov 17, 2021
582bf56
Merge branch 'main' into create-test-redis-queue
mboudreau Nov 17, 2021
c147c91
Merge branch 'main' into create-test-redis-queue
mboudreau Nov 18, 2021
d5c81cf
Merge branch 'main' into create-test-redis-queue
mboudreau Nov 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ SQS_BACKFILL_QUEUE_URL=http://127.0.0.1:4566/000000000000/test-backfill
SQS_BACKFILL_QUEUE_REGION=us-west-1
SQS_PUSH_QUEUE_URL=http://127.0.0.1:4566/000000000000/test-push
SQS_PUSH_QUEUE_REGION=us-west-1
BULL_QUEUE_PREFIX=bulltest

MICROS_AWS_REGION=us-west-1

Expand Down
1 change: 1 addition & 0 deletions src/config/env.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ export interface EnvVars {
GIT_BRANCH_NAME?: string;
GITHUB_REPO_URL: string;
DEPLOYMENT_DATE: string;
BULL_QUEUE_PREFIX?: string;

// Test Vars
ATLASSIAN_SECRET?: string;
Expand Down
2 changes: 2 additions & 0 deletions src/worker/queues.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import * as Sentry from "@sentry/node";
import statsd from "../config/statsd";
import { getLogger } from "../config/logger";
import {metricError, redisQueueMetrics} from "../config/metric-names";
import envVars from "../config/env";

const client = new Redis(getRedisInfo("client"));
const subscriber = new Redis(getRedisInfo("subscriber"));
Expand All @@ -26,6 +27,7 @@ const getQueueOptions = (timeout: number): QueueOptions => {
removeOnComplete: true,
removeOnFail: true
},
prefix: envVars.BULL_QUEUE_PREFIX,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a very simple and elegant solution.

just want to make sure that if we don't have this env var in prod we won't add any prefixes. (Cuz I am afraid if we'll start using another "queue" in prod and loose messages from the one we were using before.)

Is there any way to test it in ddev/staging? Would be great to do before merging

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the env var isn't set anywhere else but in .env.test and will just return undefined for all other environments, which will then default it.

However, that being said, I think I found a better option which is to set the key prefix for all redis keys and not just bull, so that no tests ever affects development.

Happy to deploy to staging just to be 100% sure though. One way to test this would be to start a big sync before, deploy this to staging, then make sure the sync continues afterwards.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, nvm, using redis keyPrefix didn't seem to work well with bull queues for some reason.

settings: {
// lockDuration must be greater than the timeout, so that it doesn't get processed again prematurely
lockDuration: timeout + 500
Expand Down