Tracking issue: Gitserver migration to Rendezvous hashing

[asdine]

Plan

This is the tracking issue for RFC 595.

Tracked issues

@unassigned: 31.50d

Completed: 31.50d

• (🏁 20 days ago) gitserver: Write documentation about the migration to rendezvous hashing #32324 3.00d
• (🏁 20 days ago) gitserver qa: Verify that the migration works properly #32322 5.00d
• (🏁 20 days ago) gitserver: Write a repo migration worker #32827 (PRs: #32830, #32831, #32886, #32997, #33003, #33059, #33067, #33071, #33072, #33210) 5.00d
• (🏁 20 days ago) gitserver: Instrument migration progress per shard #32468
• (🏁 20 days ago) gitserver: Create a grafana dashboard to track migration progress #32467
• (🏁 20 days ago) gitserver: Add instrumentation for clone time #32466
• (🏁 20 days ago) gitserver qa: Determine which concurrency level is the best for Cloud #32321 3.00d
• (🏁 20 days ago) gitserver: Create a table for the rendezvous migration #32313 0.50d
• (🏁 20 days ago) gitserver: Write the OOB migration to Rendezvous hashing #32314 5.00d
• (🏁 20 days ago) gitserver migration planner: Add a command to predict maximum disk usage #32316 5.00d
• (🏁 20 days ago) gitserver migration planner: Add a command to list all repositories that are going to be migrated #32317 4.00d
• (🏁 3 days ago) gitserver migration planner: Displays a static list of migration duration examples #32318 1.00d

@asdine: 5.00d

• gitserver: add GitserverReplicator worker #33072

Completed: 5.00d

• (🏁 578 days ago) gitserver: Store repository size in gitserver_repos table #32197 (PRs: #32325, #32364, #32369, #32429, #32431) 2.00d
• (🏁 577 days ago) gitserver: add GetCursor helper function #32491
• (🏁 573 days ago) gitserver: clone a repository from one instance to another #32312 (PRs: #32432, #32503) 3.00d
• (🏁 566 days ago) gitserver: add gitserver_localclone_jobs table #32830
• (🏁 565 days ago) gitserver: add GitserverLocalClone store #32886
• (🏁 564 days ago) gitserver: change RequestRepoMigrate signature #32997
• (🏁 563 days ago) db: add gitserver_localclone_jobs_with_repo_name view #33003
• (🏁 563 days ago) gitserver: add RemoveFrom method #33059
• (🏁 563 days ago) db: add queued_at column to gitserver_localclone_jobs #33067
• (🏁 563 days ago) gitserver: fix hostname in RequestRepoMigrate #33071
• (🏁 558 days ago) db: rename gitserver_localclone_jobs to gitserver_relocator_jobs #33210

@sashaostrikov: 3.00d

Completed: 3.00d

• (🏁 566 days ago) Remove global gitserver.DefaultClient #32781
• (🏁 566 days ago) gitserver: change the way gitserver client is created #32880
• (🏁 564 days ago) gitserver client: Use migration cursor to determine which instance to query #32315 (PRs: #32621, #32769, #32890) 3.00d
• (🏁 563 days ago) gitserver: simplify logic in getPinnedRepoAddr function #33074
• (🏁 553 days ago) gitserver janitor: check and insert repo size to gitserver_repos table #33000 (PRs: #33263)
• (🏁 551 days ago) gitserver: add logs to janitor and make the test less primitive in terms of repo names #33487
• (🏁 550 days ago) gitserver: add GetByNames function to gitserver store #33560
• (🏁 496 days ago) gitserver: add batched getting repos by name functionality to gitserverRepoStore  #33485
• (🏁 432 days ago) gitserver janitor: get total repo size from the database #32482

@vrto: 4.00d

Completed: 4.00d

• (🏁 508 days ago) gitserver: Add support for repository pinning #32319 (PRs: #32831, #32831) 4.00d

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🎩 Quality of life
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• 🙆 Stretch goal

[github-actions]

Heads up @jplahn @dan-mckean - the "team/repo-management" label was applied to this issue.

[dan-mckean]

Suggested weekly summary headings:

• Progress since last update
• Overall progress towards goal/completion
• Any blockers/learnings/changes to the plan since last update

[asdine]

Last week

We started the ground work:

• Storing all repositories size in the database, which will allow us to statically calculate the disk size required for each gitserver instance during the migration. gitserver: Store repository size in gitserver_repos table #32197
• Added an internal Cursor API that will be used by both the migration and the gitserver clients. This is useful to parallelize the work between us. gitserver: add GetCursor helper function #32491
• Finished the work needed to allow cloning a repo from one gitserver instance to another. Now this endpoint can be used by the migration to move things around. gitserver: clone a repository from one instance to another #32312

This week

We will work in parallel:

• @asdine: Write the migration (gitserver: Write the OOB migration to Rendezvous hashing #32314) and other satellite tasks
• @sashaostrikov: Work on the gitserver client gitserver client: Use migration cursor to determine which instance to query #32315

Progress and blockers

The work has just started last week but everything went smoothly.
The team however is not at 100% capacity.

[asdine]

Last week

• @asdine was on support the whole week
• @sashaostrikov Worked on injecting the database handle everywhere the gitserver is used. gitserver: add db connections which are needed for future instantiation of gitserver clients with db connections #32769

This week

• @asdine:
  • Break down the migration work into smaller tasks
  • Write the migration (gitserver: Write the OOB migration to Rendezvous hashing #32314)
• @sashaostrikov
  • Inject the database handle in gitserver client and replace gitserver.DefaultClient with gitserver.NewClient(dbutil.DB)
  • Use the cursor in the gitserver client gitserver client: Use migration cursor to determine which instance to query #32315:

[asdine]

Last week

• @asdine Implemented a repository replicator worker gitserver: Write a repo migration worker #32827
• @sashaostrikov Redesign gitserver client to take a database parameter and read the cursor

This week

• @asdine
  • Write the migration gitserver: Write the OOB migration to Rendezvous hashing #32314
• @sashaostrikov Ensure the gitserver_repos table contains the repo size for all repos gitserver janitor: check and insert repo size to gitserver_repos table #33000