Update categories and eligibilities for DCYF.

[richardxia]

This adds a migration that makes a large number of changes to categories and eligibilities, to match the changes requested by DCYF. Each group of categories and elgibilities is handled a little bit differently, but they roughly fall into one of the following patterns:

• Create a new category/eligibility
• Rename a category/eligibility
• Mass migrate a number of resources and services from a set of old categories/eligibilities to a new one
• Sometimes delete the old categories/eligibilities when doing said mass migration, though in some cases preserve the old ones

This was implemented as a pure data migration that does not change the schema. Since data migrations must be agnostic to the actual models, we cannot use any of the model code, and instead we construct a number of raw SQL queries. These queries have been factored into independent methods so that the main migration code can be read top to bottom with high-level method calls describing the exact changes being made.

This is supposed to follow the instructions in https://docs.google.com/spreadsheets/d/1rCvcITtUdmjOz3IzjNClbgY4UO53xGV8ix7oeeXDtKc/edit#gid=688234094, and I'd appreciate it if someone could double check my work. I've been testing locally on a dump of the production database, and I'm planning to spend some time later tonight to sanity check all my queries.

[richardxia]

One bug I discovered by manually examining the DB after applying this migration is that this will result in duplicate entries in the many-to-many join tables, since we don't have any uniqueness constraints on them. For example, because categories_services doesn't have a unique key on category_id and service_id, it's possible to insert a second row with the same category and service.

I need to alter the logic to avoid creating duplicates when possible.

[richardxia]

One other oddity I noticed is that the data in Algolia will be stale after running this migration, so we'll probably want to retrigger a mass reindexing after applying this migration.

[schroerbrian]

Wow this is great - really well-thought out how you managed the changes and check for a category’s existence or non-existence first. I don’t currently have my personal computer with me for testing locally but this code LGTM. Thanks!

[schroerbrian]

One other oddity I noticed is that the data in Algolia will be stale after running this migration, so we'll probably want to retrigger a mass reindexing after applying this migration.

Good thought. I suppose this maybe be since these records are created outside of the Resource/Service.rb models, which is where the indexing code lives

[katerina-kossler]

lgtm! had one smalll q but seemed like this would all do what we want!

[richardxia]

I'm going to do one last set of sanity checking the data resulting from this migration with my local DB, specifically focusing on the last set of changes I pushed up, but otherwise I'm planning to merge this later tonight and maybe doing a deploy tomorrow.

[richardxia]

Just finished spot-checking the data, and I think the new changes I made are working correctly. I'll leave this open for a few more hours, in case if @lgarofalo wanted to check it out, but I'll merge it regardless by the end of tonight.