Command to generate a test database, part 1 #1111
Conversation
| c_id = base_contact_id + c | ||
|
|
||
| # ensure every org gets at least one contact | ||
| org = orgs[c] if c < len(orgs) else self.random_org(orgs) |
There was a problem hiding this comment.
This is important to ensure contacts for different orgs aren't stored sequentially which wouldn't resemble a real world database, and would effect query performance when an org's contacts are all bunched at the beginning or end of a table's data.
…s get same uuids for same seed
|
This is 1,000,000 contacts across 100 orgs or each has 1,000,000 contacts? Seems like the former is probably overkill. Seems our scaling issues tend to focus around single orgs being large, so wonder if we could cut down on the build speed by just building 10 orgs instead of 100. |
| # We want a variety of large and small orgs so when allocating content like contacts and messages, we apply a | ||
| # bias toward the beginning orgs. if there are N orgs, then the amount of content the first org will be | ||
| # allocated is (1/N) ^ (1/bias). This sets the bias so that the first org will get ~50% of the content: | ||
| self.org_bias = math.log(1.0 / num_orgs, 0.5) |
There was a problem hiding this comment.
Guess I could have just read the code. :)
|
It's 1,000,000 contacts split across 100 orgs. Org generation doesn't take that much time so relatively little difference between 10 and 100 orgs. Regardless of how many orgs there are, the first org always gets ~50% of the total contacts. One thing I haven't tried yet is ditching all indexes and recreating them after, but am trying to not over-engineer things at this point. |
|
A sample of a first org in a database with total 1,000,000 contacts:
|
So far only generates non-message data: locations, orgs, users, groups, fields, labels, test contacts, contacts, contact group memberships, contact field values. Figure I'll add messages when I get onto a improving message searching performance.
The default settings generate a database with 100 orgs and 1,000,000 contacts. This takes about an hour and the resultant database is ~3.1GB.
PR includes a ~7MB dump of the AdminBoundary table in Postgres's compressed/custom format, after loading the test-data/nigeria.zip geojson file. Loading the geojson file takes half an hour but the dump loads in seconds.