feat: enrich custom companies by AmarTrebinjac · Pull Request #3311 · dailydotdev/daily-api

AmarTrebinjac · 2025-11-28T11:58:55Z

Enrich companies/schools using Claude to see if it has details on it.
Added "altName" to company table so that we can preserve both English and Native name. Can use it when searching companies (linkedIn does this too), and potentially other stuff as well.

When enriching, we will replace the user's typed in name with the "English" name. Example below I type in the Norwegian name for my school "Handelshøyskolen BI", but after enrichment, it uses the English name.

Screen.Recording.2025-11-28.at.14.57.08.mov

pulumi · 2025-11-28T11:59:31Z

🍹 The Update (preview) for dailydotdev/api/prod (at d6636c0) was successful.

Resource Changes

    Name                                                   Type                           Operation
~   vpc-native-personalized-digest-deployment              kubernetes:apps/v1:Deployment  update
~   vpc-native-daily-digest-cron                           kubernetes:batch/v1:CronJob    update
~   vpc-native-calculate-top-readers-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-generate-search-invites-cron                kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-images-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-users-cron                     kubernetes:batch/v1:CronJob    update
-   vpc-native-api-db-migration-ba1273a5                   kubernetes:batch/v1:Job        delete
+   vpc-native-api-clickhouse-migration-86905be0           kubernetes:batch/v1:Job        create
+   vpc-native-api-db-migration-86905be0                   kubernetes:batch/v1:Job        create
~   vpc-native-update-tags-str-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-generic-referral-reminder-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-temporal-deployment                         kubernetes:apps/v1:Deployment  update
~   vpc-native-clean-gifted-plus-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-sync-subscription-with-cio-cron             kubernetes:batch/v1:CronJob    update
~   vpc-native-update-current-streak-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-history-day-clickhouse-cron  kubernetes:batch/v1:CronJob    update
~   vpc-native-update-tag-recommendations-cron             kubernetes:batch/v1:CronJob    update
-   vpc-native-api-clickhouse-migration-ba1273a5           kubernetes:batch/v1:Job        delete
~   vpc-native-bg-deployment                               kubernetes:apps/v1:Deployment  update
~   vpc-native-update-source-tag-view-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-user-companies-cron            kubernetes:batch/v1:CronJob    update
~   vpc-native-update-highlighted-views-cron               kubernetes:batch/v1:CronJob    update
~   vpc-native-private-deployment                          kubernetes:apps/v1:Deployment  update
~   vpc-native-clean-stale-user-transactions-cron          kubernetes:batch/v1:CronJob    update
~   vpc-native-hourly-notification-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-update-source-public-threshold-cron         kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-cron                    kubernetes:batch/v1:CronJob    update
~   vpc-native-deployment                                  kubernetes:apps/v1:Deployment  update
~   vpc-native-post-analytics-clickhouse-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-update-trending-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-ws-deployment                               kubernetes:apps/v1:Deployment  update
~   vpc-native-validate-active-users-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-opportunities-cron             kubernetes:batch/v1:CronJob    update
~   vpc-native-update-views-cron                           kubernetes:batch/v1:CronJob    update
~   vpc-native-check-analytics-report-cron                 kubernetes:batch/v1:CronJob    update

AmarTrebinjac · 2025-11-28T12:05:14Z

 @Index('IDX_company_name_slugify', { synchronize: false })
+@Index('IDX_company_altName_trgm', { synchronize: false })
+@Index('IDX_company_altName_lower', { synchronize: false })
+@Index('IDX_company_altName_slugify', { synchronize: false })


added the same indexes as we have on name

AmarTrebinjac · 2025-11-28T13:59:35Z

+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+async function tryFetchDomain(


Not sure if this is the best way to do this, open to suggestions.

Basically some domains don't resolve without www, and some don't resolve with. Had some trouble with the Universities i tested, so the idea is to test both paths.

funny that those domains do not support automatic www redirect and so on ... so 1999

rebelchris

Should we maybe run the actual anthropic stuff via Bragi? it handles things internally better maybe.

rebelchris · 2025-12-01T08:37:15Z

+function getGoogleFaviconUrl(domain: string): string {
+  return `${GOOGLE_FAVICON_URL}?domain=${encodeURIComponent(domain)}&sz=${FAVICON_SIZE}`;
+}


We actually have this internally in API already a wrapper incase we ever wanna switch to another system.

We have /icon=url= endpoint on API that proxies to google favicons

AmarTrebinjac · 2025-12-01T08:57:54Z

Should we maybe run the actual anthropic stuff via Bragi? it handles things internally better maybe.

@rebelchris I mean we could, but I've never used Python or worked in Bragi before, so it's going to take me a minute to redo :lolsob:

capJavert

Nothing really blocking aside from using the new topic overhead, but overall some general comments on implementation to make it more reliable.

capJavert · 2025-12-01T08:50:06Z

+    domain: string;
+  };
+
+  logger.info(


this and other loggerInfo will spam a lot, let's move them to debug, also see if you really need warn, its ok if you need it for a first deployment, but after that most likely they will just spam without anyone looking at them

capJavert · 2025-12-01T08:52:41Z

+        if (attempt < DOMAIN_CHECK_RETRIES) {
+          await sleep(RETRY_DELAY_MS * attempt);
+        }


Garmr has support for retries, I would not bother in detecting what the error is, if we want to check 3 times we just set maxAttempts on garmr and do simple fetch

capJavert · 2025-12-01T08:54:16Z

+  const existingCompany = await con
+    .getRepository(Company)
+    .createQueryBuilder('company')
+    .where(':domain = ANY(company.domains)', { domain: validatedDomain })


this can be read replica I think?

capJavert · 2025-12-01T08:57:15Z

+  },
+});
+
+export const anthropicClient = process.env.ANTHROPIC_API_KEY


Does client break on initialization if API key is null, seems error prone that this variable would be null or object depending on env, can it just always be:

new AnthropicClient(process.env.ANTHROPIC_API_KEY, { garmr: garmrAnthropicService, })

capJavert · 2025-12-01T09:00:08Z

+    experience.customCompanyName &&
+    !experience.companyId
+  ) {
+    await triggerTypedEvent(logger, 'api.v1.company-enrichment', {


could we avoid adding new topic all together and just running enrichCompanyForExperience logic here?

capJavert · 2025-12-01T09:04:48Z

+  await con.getRepository(Company).save(company);
+  logger.info(
+    { companyId, name: englishName, type: companyType },
+    'Company created',
+  );
+
+  await con
+    .getRepository(UserExperience)
+    .update({ id: experienceId }, { companyId });


Do all writes in transaction so we don't get partials on errors

I'm not sure I understand. Why would we get partials on a singular update? And its not a case where we want to cancel all if one fails imo

you want to stop if any of them fail in a batch?

I don't think we need to necessarily since if one company/experience doesn't get verified/related, it doesn't really matter for the rest of them. If we stop and retry we'll have to redo API calls to the LLM for all of them again. If we get fails, wouldn't it just be better to do a "full run" at the end that picks up all the stragglers?

I am not sure we understand each other I am talking about writing everything company related in transaction, be that a batch of multiple companies (from script) or this one, to avoid writing parts of data or having incomplete data written when error happens.

You can then control how you want to handle fails, terminate transaction, or continue by catching it.

Either way up to you, try and you will see if its slow or you have issues with enrichment.

capJavert · 2025-12-01T09:05:35Z

    subscription: 'api.parse-cv-profile',
  },
+  {
+    topic: 'api.v1.company-enrichment',


if you end up using this you need to add it to streams repo

capJavert · 2025-12-01T09:07:09Z

+  const limitArgument = process.argv[2];
+  const offsetArgument = process.argv[3];
+
+  if (!limitArgument || !offsetArgument) {
+    throw new Error('limit and offset arguments are required');
+  }
+
+  const limit = +limitArgument;
+  if (Number.isNaN(limit)) {
+    throw new Error('limit argument is invalid, it should be a number');
+  }
+
+  const offset = +offsetArgument;
+  if (Number.isNaN(offset)) {
+    throw new Error('offset argument is invalid, it should be a number');
+  }


not blocking for this PR, but there is really cool parseArgs and zod pattern so you can easily validate, something for your future bin script, check importProfileFromJSON script

capJavert · 2025-12-01T09:11:15Z

+      await con
+        .getRepository(UserExperience)
+        .update({ id: experience.id }, { companyId: exactMatch.id });


Also see to do a batch in a transaction, also updating user experience can have impact on CDC so start slow and monitor

This reverts commit 414d914.

feat: enrich custom companies

30eb65c

tests

c715480

github-advanced-security AI found potential problems Nov 28, 2025

View reviewed changes

Comment thread __tests__/workers/companyEnrichment.ts Fixed

AmarTrebinjac commented Nov 28, 2025

View reviewed changes

AmarTrebinjac added 2 commits November 28, 2025 13:42

update tests

6a367e7

add script

da1b28d

AmarTrebinjac commented Nov 28, 2025

View reviewed changes

AmarTrebinjac added 2 commits November 28, 2025 15:07

update script to batch

1da7330

update slugging and check for existing companies in script

7fb6ad4

AmarTrebinjac marked this pull request as ready for review November 29, 2025 03:29

AmarTrebinjac requested review from a team and capJavert as code owners November 29, 2025 03:29

AmarTrebinjac requested review from omBratteng and rebelchris November 29, 2025 03:29

rebelchris reviewed Dec 1, 2025

View reviewed changes

capJavert reviewed Dec 1, 2025

View reviewed changes

AmarTrebinjac and others added 4 commits December 1, 2025 14:48

Merge branch 'main' into ENG-46-enrichment

72ec7c3

remove worker, logging

36cd4e0

move to env

55c205e

vars in tests

32f8134

AmarTrebinjac requested a review from capJavert December 1, 2025 23:16

AmarTrebinjac added 2 commits December 2, 2025 10:50

Merge branch 'main' into ENG-46-enrichment

638b654

Merge branch 'main' into ENG-46-enrichment

d6636c0

AmarTrebinjac merged commit 414d914 into main Dec 2, 2025
9 checks passed

AmarTrebinjac deleted the ENG-46-enrichment branch December 2, 2025 11:20

AmarTrebinjac restored the ENG-46-enrichment branch December 2, 2025 11:37

AmarTrebinjac added a commit that referenced this pull request Dec 2, 2025

Revert "feat: enrich custom companies (#3311)"

0cd23c3

This reverts commit 414d914.

AmarTrebinjac deleted the ENG-46-enrichment branch December 2, 2025 11:37

Conversation

AmarTrebinjac commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pulumi Bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Resource Changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rebelchris left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmarTrebinjac commented Dec 1, 2025

Uh oh!

capJavert left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmarTrebinjac Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

capJavert Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AmarTrebinjac commented Nov 28, 2025 •

edited

Loading

pulumi Bot commented Nov 28, 2025 •

edited

Loading

AmarTrebinjac Dec 1, 2025 •

edited

Loading

capJavert Dec 2, 2025 •

edited

Loading