K8s-based Migration

[l-monninger]

Summary

mtma is written s.t. it should be possible to simply place a container (as enabled under #143)--with the appropriate access to disk, services, and similar resources--and migrate with mtma checked-migration migrate select --all-required.

Below I propose a progressive model to begin the discussion of how this K8s-based migration could be structured.

Tip

Here are some good articles on various similar projects:

• Andrew Lock on Job-based Database Migrations in K8s
• Kapstan K8s Migrations

These are job-based. However, because of tracking features, traffic switching, and manual supervision, it may make more sense for us to place a pod with a continuously operating mtma checked-migration migrate select --all-required process, then replace this once the migration has been validated, i.e., with helm upgrade.

Note

This does not include considerations for traffic mirroring and tracking. It is assumed traffic mirroring is handled at the K8s control plane and above, while tracking would be handled within mtma.

Simple Volume-only mtma Pod

At the time of writing, it seems it may be feasible to complete the migration by simply sharing access to the persistent volume, via ReadWriteMany amongst movement, mtma, and movement-aptos pods. That is, there many not be a need for additional migration-specific services to be shared amongst these pods--only the volume and the public APIs.

This model would looks as follows:

Volume and Migration Control Plane

In the event we determine bespoke services are need to manage the migration, the model becomes slightly more complex, but still manageable through a fairly simple model of pod placement:

Important

The Migration Control Plane here does not refer to the K8s control plane under which the mtma pod is allocated. Instead it would simply refer to the bespoke services, service discovery, and special pods which may allow a migration to run gracefully.

Migrating the Current movement Network to K8s

In order for any of these K8s-based migrations to be possible, we would need to ensure the existing network and its volume are migrated to resources usable within K8s (EKS). As such, a plan for the migration of the existing EC2 volume is needed; it may look something like the below:

K8s End-to-end

In total, the e2e of the migration in K8s may take a form similar to the following:

Discussion Points

1. Does the above make sense as an initial guiding structure?
2. What can be simplified?
3. What benefits do we gain from doing this more manually?
4. To what extent can or should we automate traffic switching based on checked-migration outcomes?
5. How many times do we really expect to run the migration?

[l-monninger]

5. How many times do we really expect to run the migration?

I am guessing quite a few--more than 16 times in non-local, e2e environments. This is mainly because it will probably take a lot attempts in different environments to get it right and ultimately we can't really risk doing this without have tried it many times.

[l-monninger]

Maybe we don't have K8s for movement?

As @radupopa369 raised, it might not be so easy to introduce K8s for movement, so it could be easier to do the Movement tradition of a docker-compose based rollout.

Counterpoints

1. This likely just pushes the debt over towards movement-aptos, as currently Aptos provides helm charts which most operators use. We're right to ask whether its more costly for us to (a) upgrade movement to K8s; (b) keep the movement EC2 and docker-compose application and the K8s-based movement-aptos application but find ways to effectively move environments inbetween; or (c) create an EC2 and docker-compose application for movement-aptos. This is a good thread to try and answer that question.
2. The version of the movement network which sits within K8s shouldn't need to account for all of the usability we've added with the more recent IaC updates. To some extent, we should be fine as long as we have access to one valid replica.

[nicholasflintwillow]

Last I checked with @radupopa369, but this was a bit ago, was that the helm chart for the leader he was working on was almost done.

I think we need to switch to k8s for movement for 3 big reasons.

1. We can scale each of our services independently and easily. This was an issue we saw when trying to deploy this week where the da was taking up too much resources on the new node size and we had to break it out.
2. Direct access to the underlying nodes would be limited. This is great for security but also a problem I've noticed. We hop on the nodes and make manual edits to it that don't always make it into the IaC. This causes drift and could lead to accidentally taking down the network.
3. It requires less in house knowledge to know for our engineers and node operators when on boarding. All they would have to worry about/know is k8s instead of the bespoke stack we use to get binaries running.

The idea was to get the current nodes arch into the new IaC to help with the tech debt, then we focus on the k8s migration. We're just hitting that point now so we'll be moving onto the migration this month depending on how long it takes to clean up after the upgrade this week.

[l-monninger]

So, @mcmillennick, @radupopa369, and @sausagee, do we want to make execution of the migration contingent upon having this K8s setup?

[sausagee]

I would decouple this and do it after the L1 migration. Risk wise, the migration itself is challenging enough.

[l-monninger]

@sausagee

Which steps would you recommend decoupling?

It's more expressive and in some ways easier to do at least everything to the right of live movement in K8s (or a similar scheduler) if we can.

[l-monninger]

@areshand

Would be good to have your comments here.