Skip to content

refactor: add neverReenrich support and clean up worker structure #3837

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
skwowet merged 6 commits into from
Feb 10, 2026
Merged

refactor: add neverReenrich support and clean up worker structure #3837

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
099425d
refactor: add neverReenrich flag to prevent re-queuing one-time enric…
skwowet Feb 10, 2026
586685a
refactor: update cacheObsoleteAfterSeconds to be optional and improve…
skwowet Feb 10, 2026
3914727
refactor: make cacheObsoleteAfterSeconds optional and enhance neverRe…
skwowet Feb 10, 2026
e456986
refactor: enhance cache handling logic for enrichment sources
skwowet Feb 10, 2026
6b36d26
refactor: enforce cacheObsoleteAfterSeconds requirement for enrichmen…
skwowet Feb 10, 2026
4710c1d
refactor: reorganize member enrichment activities and schedules, remo…
skwowet Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions services/apps/members_enrichment_worker/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@
"start:debug": "CROWD_TEMPORAL_TASKQUEUE=members-enrichment SERVICE=members-enrichment-worker LOG_LEVEL=trace tsx --inspect=0.0.0.0:9232 src/main.ts",
"dev:local": "nodemon --watch src --watch ../../../libs --ext ts --exec pnpm run start:debug:local",
"dev": "nodemon --watch src --watch ../../../libs --ext ts --exec pnpm run start:debug",
"script:onboarding:local": "set -a && . ../../../../backend/.env.dist.local && . ../../../../backend/.env.override.local && set +a && SERVICE=script tsx --inspect src/bin/onboarding.ts",
"script:onboarding": "SERVICE=script tsx --inspect src/bin/onboarding.ts",
"lint": "npx eslint --ext .ts src --max-warnings=0",
"format": "npx prettier --write \"src/**/*.ts\"",
"format-check": "npx prettier --check .",
Expand Down
18 changes: 9 additions & 9 deletions services/apps/members_enrichment_worker/src/activities.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,21 @@ import {
fetchMemberDataForLLMSquashing,
findMemberEnrichmentCache,
findMemberIdentityWithTheMostActivityInPlatform,
findWhichLinkedinProfileToUseAmongScraperResult,
getEnrichmentData,
getEnrichmentInput,
getMaxConcurrentRequests,
getObsoleteSourcesOfMember,
getPriorityArray,
hasRemainingCredits,
insertMemberEnrichmentCache,
isCacheObsolete,
normalizeEnrichmentData,
refreshMemberEnrichmentMaterializedView,
squashMultipleValueAttributesWithLLM,
squashWorkExperiencesWithLLM,
touchMemberEnrichmentCacheUpdatedAt,
touchMemberEnrichmentLastTriedAt,
updateMemberEnrichmentCache,
updateMemberUsingSquashedPayload,
} from './activities/enrichment'
import {
getEnrichableMembers,
getMaxConcurrentRequests,
getMemberById,
} from './activities/getMembers'
import { refreshToken } from './activities/lf-auth0/authenticateLFAuth0'
import {
getIdentitiesExistInOtherMembers,
Expand All @@ -40,9 +33,16 @@ import {
updateIdentitySourceId,
} from './activities/lf-auth0/githubIdentities'
import {
findWhichLinkedinProfileToUseAmongScraperResult,
squashMultipleValueAttributesWithLLM,
squashWorkExperiencesWithLLM,
} from './activities/llm'
import {
getEnrichableMembers,
getMemberById,
syncMembersToOpensearch,
syncOrganizationsToOpensearch,
} from './activities/syncEnrichedData'
} from './activities/member'

export {
touchMemberEnrichmentLastTriedAt,
Expand Down
Loading