feat: Add seed people and companies data for demo environment (#2207) #2307

khakimov · 2023-11-02T04:24:36Z

This pull request adds seed data (600 companies + 1548 people) for a demo environment. The data is going to be used to populate the database for demonstration purposes.

FelixMalfait · 2023-11-02T08:52:58Z

Great! Thank you
The company data is perfect.
As for the people, I'm a little worried about the data privacy impact of including real people, I don't think we should.
Should we instead generate those names/emails once with a faker library?
Bonus (not easy): scrape something like https://generated.photos/faces and add base64 images for each contact :)

khakimov · 2023-11-03T07:36:35Z

Great! Thank you The company data is perfect. As for the people, I'm a little worried about the data privacy impact of including real people, I don't think we should. Should we instead generate those names/emails once with a faker library? Bonus (not easy): scrape something like https://generated.photos/faces and add base64 images for each contact :)

def generate_person(company_name):
    gender = random.choice(['male', 'female'])
    if gender == 'male':
        firstName = fake.first_name_male()
    else:
        firstName = fake.first_name_female()
    lastName = fake.last_name()
    random_number = secrets.token_hex(5)  # generates a 10 character long hexadecimal string
    url = get_image_url(gender)
    base_image = download_image(url)
    print(f"Saving {firstName}, {gender}...")

    person = {
        "company": company_name,
        "firstName": firstName,
        "lastName": lastName,
        "city": fake.city(),
        "email": f"{firstName.lower()}.{lastName.lower()}@example.com",
        "avatarUrl": f"data:image/jpeg;base64,{base_image}",
        "linkedinUrl": f"/in/{firstName.lower()}-{lastName.lower()}-{random_number}",
        "jobTitle": fake.job()
    }
    return person

agree about privacy, here is updated list of people_demo.json with avatars in base64, format as showed in person above. For each company generated 2 people (=1200 total)

FelixMalfait · 2023-11-03T09:17:10Z

Great!
Last round, could you:

remove "company" field from people.json (not needed as the relationship is created after)
rename people_demo to people-demo and companies_demo to companies-demo
update createDefaultPeople so that the each group of 2 people are matched with respective company, so I think it means changing companyId: companies[i] to companyId: companies[Math.floor(i / 2)]

I'll create an issue to add pagination because the app is almost crashing!

khakimov · 2023-11-03T14:27:19Z

Let me know if that looks better and if any changes needed.

Next I will look into how to create proper pipeline-demo.json based on this data and implement reset endpoint from #2207

charlesBochet · 2023-11-09T07:55:45Z

Taking a look this morning!

charlesBochet · 2023-11-09T15:45:53Z

@khakimov Thank you, we are holding a bit on this one as we are migrating companies and people models to the new schema. We will merge it once the new models are ready

charlesBochet · 2023-11-14T13:27:59Z

@khakimov, we have refactored the way we seed a workspace using the new schema.
Could you introduce a new command (similar to DataSeedTenantCommand) to seed a DemoTenant.

You'll need to your seeds into typeorm-seeds/tenant/demo
We are still thinking about how we should re-organize folders but we are sure about the seed format. Would be great to split the demo seeds into files smaller files to ease readibility

khakimov · 2023-11-20T14:46:59Z

started working on seeding data with new schema (pushed some code above already), @charlesBochet let me know if organization of files is suitable. * I will adjust people-demo.json for new format as well.

I will use DataSeedWorkspaceCommand to make DemoTenant and put together the rest of demo seeding process.

charlesBochet · 2023-11-21T22:47:00Z

@Weiko @magrinj could you take a look at these demo seeds? Would be awesome to have them to demo Twenty performances

charlesBochet · 2023-11-28T23:38:24Z

Discussed with you on Discord @khakimov, I'm available if you need a hand to complete this one!

- seedCoreSchema() - seedMetadataSchema()

charlesBochet · 2023-11-29T15:46:04Z

server/src/database/commands/data-seed-demo-workspace.command.ts

+  // TODO: get workspaceId from .env
+  workspaceId = '20202020-1c25-4d02-bf25-6aeccf7e1337';
+  dataSourceId = '20202020-7f63-47a9-b1b3-6c7290ca1337';
+  workspaceSchemaName = 'workspace_1wgvd1injqtife6y4rvfb1337';


I see that you add 1337 at the end of the name of Ids. Why not! However for workspaceSchemaName, it might not work because in some places it's computed based on workspaceId. Might lead to bugs :)

charlesBochet · 2023-11-29T15:49:40Z

server/src/database/commands/data-seed-demo-workspace.command.ts

+      await dataSource.initialize();
+
+      await seedCoreSchema(dataSource, this.workspaceId);
+      await seedMetadataSchema(


I think this won't actually work because these metadata seeds have hardcoded ids.
Could you leverage:

const createdObjectMetadata = await this.createStandardObjectsAndFieldsMetadata( dataSourceMetadata.id, workspaceId, );

instead see from workspace-manager.service.ts

Now that I think about it, what we want to do is exactly what we have in the init method from workspace-manager.service.ts.

Change in strategy:

Keep what you have to create the core entities

revert the changes to the typeorm-seeds/metadata

create a initDemo in workspace-manager.service.ts that is doing exactly what we want but instead of "prefillWorkspaceWithStandardObjects" we want "prefillWorkspaceWithDemoObjects"

This should work like a charm, sorry I did not think about it earlier

Thank you for your quick review, I will adjust accordingly!

- delete workspace - initDemo() with prefillWorkspaceWithDemoObjects() * added companies-demo.ts with data * added people-demo.ts with data

- created core/demo/ to keep modified seedCoreSchema() there - DemoSeedUserIds with new set of users and Ids

charlesBochet

This is gold!
So, next step:

revisit table performance, we really need to put back in place Implement table record virtualizer back #2728 and make sure we don't have re-renders on our table component. I have removed many to make it usable but we will need to another pass.
@lucasbordeau apollo onCompleted callback is not triggered on fetchMore, I have made a patch in the code base in useFindManyRecords but I think we can do better to remove duplicated code

Then, I think we should move dev seeds to the same place as prefills and demo seeds

charlesBochet · 2023-12-02T16:21:15Z

server/src/database/commands/data-seed-demo-workspace.command.ts

+import { EnvironmentService } from 'src/integrations/environment/environment.service';
+import { WorkspaceManagerService } from 'src/workspace/workspace-manager/workspace-manager.service';
+
+// TODO: implement dry-run


no need for dry-run for seed command

charlesBochet · 2023-12-02T17:35:56Z

front/src/modules/object-record/hooks/useFindManyRecords.ts

@@ -142,6 +142,23 @@ export const useFindManyRecords = <
                ...fetchMoreResult?.[objectMetadataItem.namePlural]?.edges,
              ]);
            }
+            onCompleted?.({


@lucasbordeau FYI

charlesBochet added the PR: awaiting review label Nov 9, 2023

charlesBochet added blocked: conflict/dependency and removed PR: awaiting review labels Nov 9, 2023

charlesBochet added PR: awaiting author and removed blocked: conflict/dependency labels Nov 14, 2023

feat: seed companies and people data

4acdfd4

khakimov force-pushed the demo-environment branch from 9fb1512 to 4acdfd4 Compare November 20, 2023 14:42

charlesBochet added PR: awaiting review and removed PR: awaiting author labels Nov 21, 2023

Merge branch 'main' into demo-environment

3cd37a2

charlesBochet added PR: awaiting author and removed PR: awaiting review labels Nov 28, 2023

charlesBochet self-assigned this Nov 28, 2023

khakimov added 2 commits November 29, 2023 12:58

Merge branch 'main' into demo-environment

d0eb850

init DataSeedDemoWorkspaceCommand to handle:

c02d07a

- seedCoreSchema() - seedMetadataSchema()

charlesBochet reviewed Nov 29, 2023

View reviewed changes

khakimov added 4 commits November 30, 2023 22:06

feature: Seed workspace with demo data

f29eba8

- delete workspace - initDemo() with prefillWorkspaceWithDemoObjects() * added companies-demo.ts with data * added people-demo.ts with data

Merge branch 'main' into demo-environment

e7fb67c

added workspaceId to seedFeatureFlags()

28b3a11

delete previous CoreSchema before seedCoreSchema

dab1021

khakimov and others added 6 commits December 1, 2023 16:41

added workspaceMemberPrefillData

f4bff4e

getDemoWorkspaces() to get DEMO_WORKSPACES from config

ce8c732

defined DemoSeedUserIds

1e9c2f9

- created core/demo/ to keep modified seedCoreSchema() there - DemoSeedUserIds with new set of users and Ids

generateOpportunities() to seed demo opportunities (limit = 50)

b2963c4

Merge branch 'main' into demo-environment

62ee523

Code review and fixes

c282ded

charlesBochet approved these changes Dec 2, 2023

View reviewed changes

charlesBochet reviewed Dec 2, 2023

View reviewed changes

Fix

efa1022

charlesBochet merged commit fd9467c into twentyhq:main Dec 2, 2023
3 of 5 checks passed

khakimov mentioned this pull request Dec 3, 2023

Provide a demo environment #2207

Closed

charlesBochet mentioned this pull request Dec 8, 2023

Release 0.2.1 #2879

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add seed people and companies data for demo environment (#2207) #2307

feat: Add seed people and companies data for demo environment (#2207) #2307

khakimov commented Nov 2, 2023

FelixMalfait commented Nov 2, 2023

khakimov commented Nov 3, 2023

FelixMalfait commented Nov 3, 2023

khakimov commented Nov 3, 2023

charlesBochet commented Nov 9, 2023

charlesBochet commented Nov 9, 2023 •

edited

charlesBochet commented Nov 14, 2023

khakimov commented Nov 20, 2023 •

edited

charlesBochet commented Nov 21, 2023

charlesBochet commented Nov 28, 2023

charlesBochet Nov 29, 2023

charlesBochet Nov 29, 2023

charlesBochet Nov 29, 2023

charlesBochet Nov 29, 2023

khakimov Nov 29, 2023

charlesBochet left a comment

charlesBochet Dec 2, 2023

charlesBochet Dec 2, 2023

feat: Add seed people and companies data for demo environment (#2207) #2307

feat: Add seed people and companies data for demo environment (#2207) #2307

Conversation

khakimov commented Nov 2, 2023

FelixMalfait commented Nov 2, 2023

khakimov commented Nov 3, 2023

FelixMalfait commented Nov 3, 2023

khakimov commented Nov 3, 2023

charlesBochet commented Nov 9, 2023

charlesBochet commented Nov 9, 2023 • edited

charlesBochet commented Nov 14, 2023

khakimov commented Nov 20, 2023 • edited

charlesBochet commented Nov 21, 2023

charlesBochet commented Nov 28, 2023

charlesBochet Nov 29, 2023

Choose a reason for hiding this comment

charlesBochet Nov 29, 2023

Choose a reason for hiding this comment

charlesBochet Nov 29, 2023

Choose a reason for hiding this comment

charlesBochet Nov 29, 2023

Choose a reason for hiding this comment

khakimov Nov 29, 2023

Choose a reason for hiding this comment

charlesBochet left a comment

Choose a reason for hiding this comment

charlesBochet Dec 2, 2023

Choose a reason for hiding this comment

charlesBochet Dec 2, 2023

Choose a reason for hiding this comment

charlesBochet commented Nov 9, 2023 •

edited

khakimov commented Nov 20, 2023 •

edited