Feature/c 867 organization enrichment by elayira · Pull Request #825 · linuxfoundation/crowd.dev

elayira · 2023-05-02T11:42:16Z

Changes proposed ✍️

What

`🤖 Generated by Copilot at 5415868`

This pull request implements the organization enrichment feature, which allows enriching the organization data of the users using the People Data Labs (PDL) API. It adds and updates the configuration, models, migrations, repositories, feature flags, message types, and services for the feature. It also adds a new worker function for the bulk organization enrichment operation, which is triggered by a service message of type OrganizationBulkEnrichMessage.

`🤖 Generated by Copilot at 5415868`

Sing, O Muse, of the mighty pull request
That enriched the data of organizations
With the secret key of PDL, the best
Of all the APIs of the nations.

Why

How

`🤖 Generated by Copilot at 5415868`

Add a new feature for enriching organization information using the People Data Labs (PDL) API (link, link, link, link, link, link, link, link, link)
Define a new interface OrganizationEnrichmentConfiguration for the configuration object for the feature in configTypes.ts (link)
Export the ORGANIZATION_ENRICHMENT_CONFIG object from index.ts in the config folder, which assigns the apiKey property based on the KUBE_MODE environment variable (link, link)
Define a new property ORGANIZATION_ENRICHMENT in the PLAN_LIMITS object in isFeatureEnabled.ts, which sets the limit of organization enrichment operations per month for each plan (link)
Define a new type OrganizationBulkEnrichMessage in messageTypes.ts, which holds the service and tenant id for the bulk organization enrichment operation (link)
Define a new class OrganizationEnrichmentService in organizationEnrichmentService.ts, which provides the logic for the feature, such as validating the plan limit, fetching the data from the PDL API, and updating the database models (link)
Define some types for the feature in organizationEnrichmentTypes.ts, such as OrganizationEnrichmentData, OrganizationEnrichmentResponse, and OrganizationEnrichmentError (link)
Define a new property ORGANIZATION_ENRICHMENT in the FeatureFlag enum and a new property ORGANIZATION_ENRICHMENT_COUNT in the FeatureFlagRedisKey enum in common.ts (link, link)
Add a new function BulkorganizationEnrichmentWorker in bulkOrganizationEnrichmentWorker.ts, which performs the bulk organization enrichment operation for a given tenant using the OrganizationEnrichmentService class (link)
Add a new case to the switch statement in the workerFactory function in workerFactory.ts, which handles the 'enrich-organizations' service message and calls the BulkorganizationEnrichmentWorker function (link, link, link)
Add some new columns to the organizations and organizationCaches tables in the database, which store the data obtained from the PDL API, such as lastEnrichedAt, employeeCountByCountry, type, ticker, headline, profiles, naics, industry, and founded (link, link, link, link)
- Add the corresponding migration files to the database migrations folder, which drop some unused columns and add the new columns (link, link, link, link)
- Add the corresponding properties to the Organization and OrganizationCache models in the database models folder, which have the appropriate data types and constraints (link, link)
Add a new static method bulkUpdate to the OrganizationRepository and OrganizationCacheRepository classes in the database repositories folder, which allow updating multiple organization instances in bulk using the bulkCreate method with the updateOnDuplicate and returning options (link, link)

Checklist ✅

Label appropriately with Feature, Improvement, or Bug.
Add screehshots to the PR description for relevant FE changes
New backend functionality has been unit-tested.
API documentation has been updated (if necessary) (see docs on API documentation).
Quality standards are met.

joanreyero

Great job on this! 💪🏼 It's starting to get there.

I left a few comments of things that should be changed, and some questions 🙂

joanreyero · 2023-05-05T09:16:03Z

+    FROM "tenants"
+    WHERE tenants."plan" IN ('Growth')
+      OR (tenants."isTrialPlan" is true AND tenants."plan" = 'Growth')
+    ;


Let's try to keep all the DB queries in the repository later

joanreyero · 2023-05-05T09:22:38Z

+        type: DataTypes.ARRAY(DataTypes.TEXT),
+        allowNull: true,
+      },
+      headline: {


What is this? And how is it different from the description? If it's semantically the same (or very similar), we should just use description.

Could you also add comments for all the new fields that are not obvious? Like the ones where I left a comment, and naics, ticker, type...

The headline field according to PDL is a piece of brief information about the company. More detailed info can be found in the company's summary. This is what I am using to update our description field. naics is the company's classification according to the NAIC standard. ticker is the company's trading symbol. And type is the kind of entity the company is, ex an NGO. I have commented on the fields that need it.

joanreyero · 2023-05-05T09:23:39Z

+        type: DataTypes.TEXT,
+        allowNull: true,
+      },
+      size: {


What is this? And how is it different from employees? If it's semantically the same (or very similar), we should use employeeCount.

employees is the number of workers in the company. size is the well, size of the company that's small, medium, large, etc. Size is represented as a range.

joanreyero · 2023-05-05T09:24:16Z

+        type: DataTypes.ARRAY(DataTypes.JSONB),
+        allowNull: true,
+      },
+      profiles: {


How does this differ from github, linkedin, crunchbase etc?

The profile holds all the social networks a company has. It is designed to hold multiple links to the same network. It is part of the requirement in the task document.

joanreyero · 2023-05-05T09:25:54Z

+    const isValid = new Set(data.filter((org) => org.id).map((org) => org.id)).size !== data.length
+    if (isValid) return [] as T
+
+    const orgs = await options.database.organization.bulkCreate(data, {


I might be misunderstanding something, but the function name is bulkUpdate and the operation here is bulkCreate. Is this correct? If so, please add a comment explaining why 🙂

bulkCreate can be used to create/update bulk. It updates on conflict with primary key or unique field when the updateOnDuplicate args is used. Comments have been added to the code to reflect this behavior.

joanreyero · 2023-05-05T09:29:00Z

+
+  async queryTenancyOrganizations(): Promise<IEnrichableOrganization[]> {
+    const options = await SequelizeRepository.getDefaultIRepositoryOptions()
+    const query = `


Same as before, let's keep queries in the repo layer. Feel free to use organizations repo.

joanreyero · 2023-05-05T09:39:11Z

+        },
+      },
+    )
+    return orgs.map(org => this.selectFieldsForEnrichment(org))


From PDL's docs it looks like we obtain:

employee_count (we should have this as employees)

location: it's an object with several interesting fields. I think we should store all of them as a JSONB. Except for the GEO code. Let's make a new column for it.

We are not currently updating employees with employee_count from PDL as it's not in the task requirement document.

…n-enrichment-qa-1351-prioritize-organizations-website

…n-enrichment-qa-1351-always-fill-lastEnriched

…n-enrichment-qa-1349-fill-employees-from-employeesByCountry

…d' into feature/c-867-organization-enrichment

…loyeesByCountry' into feature/c-867-organization-enrichment

elayira requested a review from epipav May 2, 2023 11:42

elayira added the Feature Created by Linear-GitHub Sync label May 3, 2023

elayira requested a review from joanreyero May 3, 2023 15:29

joanreyero requested changes May 5, 2023

View reviewed changes

elayira requested a review from joanreyero May 6, 2023 01:33

elayira marked this pull request as ready for review May 8, 2023 00:17

elayira force-pushed the feature/c-867-organization-enrichment branch from 816cab4 to 70db9ad Compare May 14, 2023 18:31

elayira changed the base branch from main to staging/cycle-25 May 14, 2023 20:03

elayira changed the base branch from staging/cycle-25 to main May 14, 2023 20:07

elayira force-pushed the feature/c-867-organization-enrichment branch from b27eb60 to bc6c7ce Compare May 14, 2023 20:19

feat: added organization enrichment

fee472b

elayira force-pushed the feature/c-867-organization-enrichment branch from bc6c7ce to fee472b Compare May 14, 2023 20:24

Merge branch 'main' into feature/c-867-organization-enrichment

89bfd43

elayira changed the base branch from main to staging/cycle-25 May 14, 2023 20:36

elayira changed the base branch from staging/cycle-25 to main May 14, 2023 20:40

elayira added 15 commits May 14, 2023 21:49

improv: flag every selected orgs as enriched

0121d5b

improv: compute missing employee count from employees by country

2be5b6c

improv: priotize orgs with website

918a26a

fixed failing test

20c6bbb

fixed failing test

9384c4b

fixed styling

0e43d1d

fixed failing test

9755686

fixed failing test

3f7a7fd

Merge branch 'feature/c-867-organization-enrichment' into organizatio…

ca6247b

…n-enrichment-qa-1351-prioritize-organizations-website

Merge branch 'feature/c-867-organization-enrichment' into organizatio…

914b3a5

…n-enrichment-qa-1351-always-fill-lastEnriched

Merge branch 'feature/c-867-organization-enrichment' into organizatio…

366c2fb

…n-enrichment-qa-1349-fill-employees-from-employeesByCountry

Merge branch 'organization-enrichment-qa-1351-always-fill-lastEnriche…

b444666

…d' into feature/c-867-organization-enrichment

Merge branch 'organization-enrichment-qa-1349-fill-employees-from-emp…

d0d2096

…loyeesByCountry' into feature/c-867-organization-enrichment

Merge branch 'main' into feature/c-867-organization-enrichment

c08f2bf

misc

1e72a2c

elayira added 5 commits May 15, 2023 11:21

misc

22771c6

misc

d8401bf

set cron to daily

b31b90d

fixed description

d96956e

format

4aadac2

joanagmaia merged commit 789faf8 into main May 16, 2023

joanagmaia deleted the feature/c-867-organization-enrichment branch May 16, 2023 09:20

Conversation

elayira commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed ✍️

What

🤖 Generated by Copilot at 5415868

🤖 Generated by Copilot at 5415868

Why

How

🤖 Generated by Copilot at 5415868

Checklist ✅

Uh oh!

joanreyero left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elayira May 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

elayira commented May 2, 2023 •

edited

Loading

`🤖 Generated by Copilot at 5415868`

`🤖 Generated by Copilot at 5415868`

`🤖 Generated by Copilot at 5415868`

elayira May 6, 2023 •

edited

Loading