Allow data sync to safely append new changes without recreating entire db every time

[jps327]

Summary

Finally, fixed the data ingestion path so that it can reliably append new rows and update records in elastic search (and delete things that have been removed from socrata!). In the past our only option was to destroy the entire db and recreate everything from scratch. Every time. Which would take hours in production. This should finally allow us to set a cron job to actually keep data in scout up-to-date.

Also added lots of comments because the database ingestion path (in the portal-sync module and in search.service) is difficult to follow and is due a massive refactor one day.

These changes require a postgres db migration unsupported by TypeORM, since we had to change the dataset_column id to be a string, and to cascade on dataset deletion. These are the postgres commands to safely migrate the tables without loss of data:

ALTER TABLE dataset_column ALTER COLUMN id TYPE character varying;
ALTER TABLE dataset_column ALTER id DROP default;
UPDATE dataset_column SET id = "datasetId" || '__' || field;
REINDEX index "PK_203534c31c4e07c6622823cfe7f";

ALTER TABLE dataset_column DROP CONSTRAINT "FK_9e07cbe18002d6ac9f5acc33109";

ALTER TABLE dataset_column ADD CONSTRAINT "FK_9e07cbe18002d6ac9f5acc33109" FOREIGN KEY ("datasetId") REFERENCES "public"."dataset"("id") ON DELETE CASCADE;

Screenshots or Videos (if applicable)

Entirely backend, no screenshots.

Related Issues

Closes #276

Test Plan

Please describe the exact steps you followed to test your change. Be as clear as possible so a reviewer can follow the same steps. List what UI interactions are needed, or if there are automated tests then list what should be run. For example:

1. Run yarn sync-database-dev and open localhost:9200/dataset_index while it runs. You should notice that the elasticsearch index has not been deleted.
2. Open package.json and change the LAST_METADATA_UPDATE_DATE env var in the seed-database-dev command to see how we only update datasets that have had recent metadata changes (this works thanks to Socrata actually tracking when the last metadata update of each dataset is, and we keep track of that in our db as well).

Checklist Before Requesting a Review

• I have performed a self-review of my code
• My code follows the Style Guidelines and Best Practices outlined in the project wiki
• I have commented my code, particularly in hard-to-understand areas
• I have made changes to the documentation, if applicable
• My change generates no new warnings or failed tests
• If it is a core feature, I have added thorough tests
• I have implemented analytics, if applicable

[indraneel]

would this need to be manually updated later?

[jps327]

This would be overridden via a environment variable when we run the data ingestion command (yarn seed-database-full). The cron job we set will set this environment variable correctly whenever we run this on a weekly basis

[indraneel]

lgtm

• i don't understand why the id is now a String rather than an int
• not sure i followed all the db ingestion code but appreciate the comments