Announcement: Repository Consolidation Timeline

[jaredpar]

As we announced earlier we are planning on consolidating some of the repositories in the dotnet org. Our planning has reached a point where we have a schedule for the coreclr, corefx and core-setup moves into dotnet/runtime that we want to share out with the community.

November 13th

We’ll move all changes from the original repositories into dotnet/runtime up to 5PM on November 13th PST (1AM November 14th UTC). We’ll try to help as many pull requests as possible get merged by then. At that point, if there’s any pull requests still open we’ll have to close them. If you’d still like to continue those pull requests, we’d encourage you to bring them to dotnet/runtime in a new pull request.

The repositories themselves will be effectively archived at this point. The state of the “master” branch will be recorded with a tag, named “master-archive”, but the branch itself will be deleted. The default branch for the repositories will be named “archive” and it will be a single commit with a README.md and CONTRIBUTING.md file pointing to our dotnet/runtime repository.

The repositories will remain active for servicing changes to .NET Core 3.1 and earlier hence we will not be using the GitHub archive capability.

November 22nd

The dotnet/runtime repository will be made public and available for community contribution. Even though the repository will be created on November 13th it will take several days to get it back into a working order: fixing up our build scripts, recreating our Azure Dev Ops build definitions, etc … Until those tasks are completed it will not be possible to accept pull request and hence the repository will remain private. Once we are in a state that pull requests can be merged again the repository will be made public.

Our expectation is that will occur on November 22nd. If it is ready sooner it will be made public sooner. If the work takes longer than we planned then we will add an update to this announcement with a new expected date.

==== as of November 27th, we are currently here ====

December 1st-2nd (Delayed)

Currently delayed, will update with a new date once it's known

All issues, open and closed, will be migrated from corefx, coreclr and core-setup into dotnet/runtime. This will be using GitHub’s existing issue transfer feature in a bulk migration. This means all of the existing issue links will continue to function via redirects.

This does mean though that labels will not transfer with the issues. Labels will be re-applied as a post processing step by our engineering team once the issue migration completes.

Even though our issues won’t be fully migrated until this time we’d like the community to begin filing issues on dotnet/runtime as soon as it’s public rather than continuing to file issues on the original repositories.

Migrating Commits

The dotnet/runtime repository will be a new commit history from the original repositories. We are using this consolidation as an opportunity to clean up our history with the goal of having a cleaner, smaller history as a starting point. This means commits will be rewritten in the following ways as they are migrated to dotnet/runtime:

1. Author information, contributor information, changed file list and time stamps from the original commit will be preserved.
2. Links to issues, pull requests or commits in the repository using GitHub short links will be rewritten so they continue to point the original repository.
3. Every commit will be appended with a link to the original commit it was mapped from.

The actual contents of the commit though will be updated to match the new directory layout of the dotnet/runtime repository.

[AaronRobinsonMSFT]

@jaredpar I am going to pin this issue since it is so impacting to the repo and community.

[huoyaoyuan]

up to 5PM PST on November 13th.

Could you add a UTC time? Not everyone are familiar with offset of PST.

[jaredpar]

@huoyaoyuan done

[gfoidl]

aspnet/Extensions is also affected (when I read correct).
Please add an announcement over there, as there's no such info so far -- only in the dotnet-org.
Or will aspnet/Extensions brought in later?

[stephentoub]

Or will aspnet/Extensions brought in later?

When and where its components move is still being investigated. It is not part of this initial consolidation.

[gfoidl]

All issues, open and closed, will be migrated from corefx, coreclr and core-setup into dotnet/runtime.

Will the issue-subscriptions (email notifications) also be transfered / migrated?

[danmoseley]

Will the issue-subscriptions (email notifications) also be transfered / migrated?

We will be using Github's regular issue transfer feature. @jeffschwMSFT was going to check whether it does this, and if not, what other options we have.

[Eilon]

I do believe notification settings are preserved (if you were watching an issue, you're still watching it, and if not, then not).

[bencyoung]

Has this been delayed? Changes still seem to be going in?

[stephentoub]

Has this been delayed? Changes still seem to be going in?

It's on track. As of Thu morning, we are no longer merging changes to master in coreclr, corefx, or core-setup.

Where do you see changes still going in?

[bencyoung]

@stephentoub No worries, was just being nosy really, as I like to follow the changes. Could have just been a timezone thing :)

[deinok]

@stephentoub Seems like its been delayed the publication of the runtime repo. Can we have an update on the expected date?

[jaredpar]

@deinok sorry for not updating already. The plan is to open the repository Monday, 11/25. There were just a few more items we wanted to finish up first.

Appreciate everyone’s patience here. Definitely excited about this going public.

[deinok]

Thanks for the update. And of course we are excited to see the final result, but we are okey waiting for it as we know that this migration is not easy

[jaredpar]

Working on finishing one last document then we'll make it public

[AustinWise]

If you already have coreclr, corefx, and core-setup cloned, you can speed up your download of the runtime repo by using the --reference argument:

git clone --dissociate --reference coreclr --reference corefx --reference core-setup https://github.com/dotnet/runtime.git

This reduced the download size from 218mb to 35mb for me.

The downside to doing this is the git object database seems to take twice as much space (500MB instead of the 220 MB on one of my machines). To reclaim some of that space delta, you can run:

git repack -Afd

So unless you are severely bandwidth limited, cloning normally is probably easier and faster.

[danmoseley]

@AustinWise that's interesting, do you know how it reduces the download? the objects cannot be reused, as the SHA's changed, right?

[AustinWise]

@danmosemsft I was also surprised that it worked, given that all the commits have been rewritten. Most of the files and directories are unchanged from the source repos. So it appears the algorithm Git uses to determine the minimal set of objects to transfer does not require having any commits in common.

[jaredpar]

Note: the issue migration has been delayed due to a miscommunication with GitHub (our fault). Once we get a new date for the migration I'll update the schedule.

[danmoseley]

The miscommunication was our fault, not Github's (who have been awesome btw)

[am11]

The consolidating has costed 17.8 K, 12.3 K, 2.4 K, 1.3 K and 431 repo stars (a measure of popularity in GitHub ecosystem) from corefx, coreclr, corert, aspnet/Extensions and core-setup respectively. 😭

[am11]

Indeed, I just wanted to point out if anything was missed, it's that. Maybe the migration tool supports this metadata and it was simply missed?

[markusschaber]

@danmosemsft I was also surprised that it worked, given that all the commits have been rewritten. Most of the files and directories are unchanged from the source repos. So it appears the algorithm Git uses to determine the minimal set of objects to transfer does not require having any commits in common.

Git has several layers, the lowest layer is kinda contend based adressing (blobs addressed by their hash). This is used for both metadata, and file contents.
So even if the commits cannot be reused, the individual file contents can.

See https://git-scm.com/book/en/v2/Git-Internals-Git-Objects and surrounding chapters for more details.

[markusschaber]

Cloning the runtime repo gives me:

warning: unable to access 'src/installer/test/Assets/TestProjects/AppWithSubDirs/Sentence/This is a really, really, really, really, really, really, really, really, really, really, really, really, really, really long file name for punctuation/.gitattributes': Filename too long

[gfoidl]

@markusschaber see https://github.com/dotnet/runtime/blob/master/docs/workflow/windows-requirements.md#enable-long-paths

[markusschaber]

@gfoidl thanks, should have known that myself. :-)

[am11]

This is a really, really, really, really, really, really, really, really, really, really, really, really, really, really long file name for punctuation

This particular directory is being removed by #373.

[danmoseley]

Indeed, I just wanted to point out if anything was missed, it's that. Maybe the migration tool supports this metadata and it was simply missed?

The migration tool is just scripts we wrote. So we can’t migrate stars. I hope the community stars us again!

[am11]

The state of the “master” branch will be recorded with a tag, named “master-archive”

tags were named master, instead of master-archive:

• https://github.com/dotnet/core-setup/releases/tag/master
• https://github.com/dotnet/corefx/releases/tag/master
• https://github.com/dotnet/coreclr/releases/tag/master

[danmoseley]

@ViktorHofer

[ViktorHofer]

After the "master" branch is deleted, a tag will be created with the same name "master", pointing to the same commit. This avoids 404s for some URLs referencing "master", most critically the license.txt links used by some .NET Core packages on the NuGet Gallery.

This was done intentionally. We have two new tags per repo: master and master-archive which are identical.

[danmoseley]

This has fulfilled its purpose.