Migrate keywords data structure to Postgres ARRAY type

[miketheman]

This will be taken apart and shipped in parts.

🚨 Note: Data migration contained within

Does not drop existing column, but removes references to it.

Once merged, no new Releases will populate their keywords column, only the keywords_array column will be filled.

[miketheman]

I thought about making this a nested comprehension to save a variable, but figured that might hamper readability so did this instead.

[miketheman]

This is the data migration I'm a little wary about - since it'll need to update somewhere around 4M records (depending on the WHERE clause in prod). Any timing estimates that could be run on TestPyPI DB would be helpful to gauge production impact.

[dstufft]

We have a statement timeout and lock timeout in migrations to prevent long running migrations from blocking the production database for a long period of time (without the migration explicitly opting into that by setting the statement timeout to something else). See:

warehouse/warehouse/migrations/env.py

Lines 53 to 54 in 61a14ce

	connection.execute("SET statement_timeout = 5000")
	connection.execute("SET lock_timeout = 4000")

In the past what we've done is "break" Alembic's migration isolation by chunking up the data migration and manually calling COMMIT in the migration, see:

warehouse/warehouse/migrations/versions/4490777c984f_migrate_existing_data_for_release_is_.py

Lines 28 to 58 in 61a14ce

	def _get_num_rows(conn):
	return list(
	conn.execute(
	sa.text("SELECT COUNT(id) FROM releases WHERE is_prerelease IS NULL")
	)
	)[0][0]
	def upgrade():
	conn = op.get_bind()
	conn.execute("SET statement_timeout = 60000")
	total_rows = _get_num_rows(conn)
	max_loops = total_rows / 100000 * 2
	loops = 0
	while _get_num_rows(conn) > 0 and loops < max_loops:
	loops += 1
	conn.execute(
	sa.text(
	"""
	UPDATE releases
	SET is_prerelease = pep440_is_prerelease(version)
	WHERE id IN (
	SELECT id
	FROM releases
	WHERE is_prerelease IS NULL
	LIMIT 100000
	)
	"""
	)
	)
	conn.execute("COMMIT")

[miketheman]

Perfect, thanks for the example.

[dstufft]

I haven't read the entire PR yet, but you probably want this split over multiple deploys, currently what's going to happen is:

1. Column will be added, data migration will be run.
2. New deployment will occur with the new code.
3. Old deployment will be spun down, shutting down old code.

The important bit of that is that the old code will still be running after the data migration has completed, so anything uploaded in that window of time via the old code will have keywords populated, but not keywords_array.

So I would refactor this into multiple PRs, that we can deploy in separate deployments:

1. Adds the keywords_array column, sets up the upload code to write both columns.
2. Does the data migration, guarding not to run against rows that (1) would have already written, sets up the code to only use the keywords_array column.
   • Optional: This could also drop the keywords column from the code, but not from the database, making SQLAlchemy completely ignore that column.
3. If the keywords column has been dropped from the code, drop it from the database as well.

[miketheman]

probably want this split over multiple deploys

Yep! Going to dismantle and rebuild, now that all the pieces are known.