Removal of the ingestion server

[stacimc]

Start Date	ETA	Project Lead	Actual Ship Date
2024-03-14		@stacimc	TBD

Description

We presently orchestrate our data refresh process with Airflow, but the operations themselves occur on a bespoke ingestion server that’s difficult to maintain, troublesome to deploy, and not nearly as robust to failures.

For this project, we will move the operational pieces of the data refresh into Airflow.

Documents

• Project proposal
  
• Implementation Plan

Issues

• Write Project Proposal for ingestion server removal #3935
• Implementation Plan for Ingestion Server removal #3981
• Create the new data refresh DAG factory and move initial steps into Airflow #4146
• Implement new catalog indexer-worker #4147
• https://github.com/WordPress/openverse-infrastructure/issues/849
• Implement local distributed reindexing #4148
• Add remaining steps to the Data refresh DAGs #4149
• Deploy catalog and new indexer workers #4150
• Update create_filtered_index DAGs to remove ingestion server dependency #4151
• Remove the recreate_full_staging_index DAG #4152
• Update the load_sample_data scripts to remove ingestion server usage #4153
• Remove ingestion server code #4154
• https://github.com/WordPress/openverse-infrastructure/issues/850

Milestones

• Removal of the data refresh server

Prior Art

[stacimc]

Adding myself as project lead at least for now. As discussed at the priority meeting, we will move forward with making the proposal and at least starting work on the implementation plan, with the understanding that implementation may not move forward if this project is discovered to be more complicated than we believe it will be.

[stacimc]

Updated the language from "data refresh server" to "ingestion server" to reflect the current name of the service, in order to be less confusing. We had experimented with renaming the ingestion server in some places in documentation; if this project proceeds we won't have to worry about this confusion anymore regardless :)

Work on the project proposal is underway, as is some early investigation into the feasibility of the project.

[openverse-bot]

Hi @stacimc, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[stacimc]

The project proposal was approved, and the IP is awaiting its second approval.

[stacimc]

The implementation plan has been merged and approved, and issues created under the milestone.

[zackkrida]

@stacimc I've moved this to "On Hold" while we wait to determine how to prioritize it; whether we start work on this project or aim to complete others first.

[stacimc]

After discussing with @zackkrida we've decided to move this into In Progress and kick off implementation. I'll be picking up the first issue today 🥳

[openverse-bot]

Hi @stacimc, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[stacimc]

This project is underway. The first major PR is up for review, with work in progress on the indexer worker image. There is also considerable progress on the infrastructure side with preparing for the indexer workers.

[stacimc]

Noting that https://github.com/WordPress/openverse-infrastructure/pull/871 has been merged and the indexer worker pools are now available!

[openverse-bot]

Hi @stacimc, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[stacimc]

Progress was delayed due to AFK. The indexer worker has been drafted here and will be up for review this week.

[openverse-bot]

Hi @stacimc, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[stacimc]

The indexer worker is fully implemented and we're moving on to the last few implementation issues. One additional issue was added to the milestone after discussion on #4464.

[stacimc]

After a conversation with @sarayourfriend yesterday, I'm considering removing the use of the autoscaling group and going back to a plan very similar to my original proposal in the IP for this project, allowing Airflow to directly manage the EC2 instances. The ASG has caused a few problems with error handling and, as noted in inline comments in the PR for implementing the distributed reindex, with retrying individual workers (one of the stated goals of the project, and a situation we've run into in recent memory).

This is not yet definitive and I will make a PR to change the implementation plan since this is a big enough change/I want to get approval. However I'll note that this would not invalidate any work that's been done so far, except for a few in-progress changes in the linked PR (which need work either way!), and removing just the ASG on the infrastructure side.

[sarayourfriend]

Great! I'll look out for the IP change (please ping me there) and will get the infra side ready for you, it should be a simple enough change, with some room for a small refactor I've been wanting to do for a while now (extract the launch template and security group creation out into a separate module, so that one-off instances like the bastion can use it instead of the old user-data approach. Anyway, it should be a quick one to implement on the infra side and unblock live testing with staging.

[openverse-bot]

Hi @stacimc, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.