Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Great Project Restructuring of 2018 #2534

Closed
vito opened this issue Aug 27, 2018 · 13 comments
Closed

The Great Project Restructuring of 2018 #2534

vito opened this issue Aug 27, 2018 · 13 comments

Comments

@vito
Copy link
Member

@vito vito commented Aug 27, 2018

Context: https://discuss.concourse-ci.org/t/psa-the-great-code-restructing-of-2018/543

  • Automate creation of "one big repo"
  • Get a proof-of-concept going using Go 1.11 modules
  • Ship 4.1.0 4.2.1
  • Prep workflow around "one big repo"
    • Design pipeline for new structure
      • Figure out how concourse-bosh-release should be built (solution: auto-bump concourse blob)
    • Create a really nice README.md (finally)
    • Preserve CODE_OF_CONDUCT.md, LICENSE.md, NOTICE.md (update attribution header? contact legal)
    • Create a new CONTRIBUTING.md for the newer (hopefully much much simpler) workflow
    • Create a better issue template
      • Instruct users to open "ops" issues on deployment-specific repo (try to limit concourse/concourse to actual bugs and feature requests)
  • Create concourse-bosh-release repo containing the current content of concourse/concourse
  • Clean up all stale branches under concourse/concourse
    • Migrate any ones we still need to concourse-bosh-release
  • Create "one big repo" based on prior automation, this time "for real"
  • Push the "one big repo" as a merge (with distinct parents) to the concourse/concourse master branch.
  • Archive atc, fly, go-concourse, tsa, web, testflight, ...
  • Restructure "one big repo"; look to other "one big repos" for patterns (e.g. pkg/..., cmd/..., web/...) (long-term)
@vito vito added the enhancement label Aug 27, 2018
@vito vito self-assigned this Aug 27, 2018
@jama22 jama22 added the blocked label Aug 27, 2018
@jama22

This comment has been minimized.

Copy link
Member

@jama22 jama22 commented Aug 27, 2018

"blocked" on a release....but @vito will keep working on READMEs and what-not

@aegershman

This comment has been minimized.

Copy link

@aegershman aegershman commented Aug 28, 2018

airplane

I just want to tell you both good luck. We're all counting on you.
(and thank you for your OSS work. we the community genuinely appreciate it 👍you're helping make our lives easier and better)

vito added a commit to concourse/atc that referenced this issue Aug 30, 2018
this isn't yet the source of truth, and will be made redundant by the
upcoming 'one big repo' (concourse/concourse#2534)
@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Aug 31, 2018

@aegershman Hah, thanks!


I just realized we're in a bit of a pickle here. The original plan was to just push -f over the concourse repo, and this is so that we can keep all the existing issues here, since they can't be moved.

But, there are also GitHub releases on the concourse repo, and all of our GitHub releases contain all the artifacts for previous versions of Concourse. If I push -f over master of concourse/concourse, all the old tags will go away, and as a result all existing releases will be turned into draft releases and unpublished. We would have to go back and manually re-add them, but all their tags wouldn't even point to anything useful, because the history is completely new and contains the sum of all the different repos, and nothing related to the release itself. I guess the tags would still be present on the concourse-bosh-release repo if we really want to see the code for a given version.

I could rename the repo instead, but that would in turn move all the existing issues to the concourse-bosh-release repo. This maintains the history much better and feels like a smoother transition, but all the issues would be in the wrong place. :/ And all the existing links between issues would break as soon as we add the one big concourse repo and make the name active again.

So we need to pick our poison: lose all release history but preserve tags on concourse-bosh-release, or rename the repo and break GitHub issue links and end up with all the GitHub issues on the wrong repo. Maybe we could also ask GitHub for help here? :/

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Sep 4, 2018

Hmm, actually we might be able to get away with this. When you push -f over master, that'll blow away master with a whole new history, but all the old tags are still there, keeping the old refs and commit history alive. The GitHub releases also stick around, with their artifacts and everything.

Side note: even with all the old tags keeping the history around, the resulting concourse repo is still pretty small. The "one big repo" with everything stripped down was ~26MB, and the push -fed repo that still contains tags and such is ~31MB. Not bad!

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Sep 4, 2018

Other things I'm thinking about while tackling this:

  • We should stop checking in compiled assets like bindata.go and elm.js.min. Repository size is a concern now that we've got everything in here together. As part of the "one-big-repo" creation script I've used BFG to clean up the history, to great effect.
    • We should switch from go-bindata to something that's still maintained. statik looks good.
  • (#2574) For the Linux developer workflow, we should make it easier to go from 0 to a fully working Garden-runC Linux worker.
    • Right now we directly import guardiancmd and embed it in the binary go-flags struct. This is kinda cool but is also tightly coupled and means we have to hoist around all of its binary dependencies. There is a self-contained gdn binary now (which hoists around and auto-wires its own dependencies) - should we just use that instead? (opened #2574 for this)
  • (#2576) The build process for the concourse binary is fairly obtuse and depends on the BOSH release for embedding resource packages directly into the binary. I think we should probably change this, but I'm not sure what to do yet. It probably makes more sense to just ship the binary alongside the resources rather than embedding it.
@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Sep 5, 2018

More questions as I discover all the ways the BOSH release workflow has worked its tendrils into our process. For fear of getting trapped in analysis paralysis, I'm at least going to write them down before coming up with answers. I guess I'll just keep editing this.

  • (#2575) When and how will we build Elm and other static assets in to the binary/BOSH release, if we're not checking it in?
  • (#2575) How do we ensure the BOSH release can build without needing to fetch Go dependencies? (Can we vendor prior to building the release and convince the bosh CLI that the tree isn't dirty somehow?)
  • (#2575) Would all of the above be easier if we just had the BOSH release be a wrapper for the concourse binary? Would the loss in flexibility be worth it? What other benefits might there be?
  • (#2576) Is it time to just start versioning and shipping the core resources properly?
    • Is this also a good time to 'trim down' the core set of supported resource types?
  • How do we cut over to this process without stopping the world and making it impossible to deliver for a while?
    • I don't think you do.
@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Sep 5, 2018

I've started switching everything over to packr: https://github.com/gobuffalo/packr

It's a lot better suited than the previously-mentioned statik package because it lets us have an arbitrary number of statically embedded things, rather than just the one single package that statik provides. It also works really well for local development; it'll just read from disk, and then in our pipeline we just run packr to snapshot the state and embed it right in the binary. Neat!

vito added a commit that referenced this issue Sep 5, 2018
#2534

Submodule src/github.com/concourse/atc f033ea706..3085d6430:
  > switch from go-bindata to packr
Submodule src/github.com/gobuffalo/packr 000000000...6c8156ee3 (new submodule)
Submodule src/github.com/inconshreveable/mousetrap 000000000...76626ae9c (new submodule)
Submodule src/github.com/pkg/errors 000000000...816c90855 (new submodule)
Submodule src/github.com/spf13/cobra 000000000...99dc12355 (new submodule)
Submodule src/github.com/spf13/pflag 000000000...298182f68 (new submodule)
Submodule src/golang.org/x/sync 000000000...1d60e4601 (new submodule)
@jchesterpivotal

This comment has been minimized.

Copy link
Contributor

@jchesterpivotal jchesterpivotal commented Sep 9, 2018

Force-push (no, really) the "one big repo" over concourse/concourse master branch.

It's actually possible to merge repos in a way that preserves their independent histories using --allow-unrelated-histories. The outcome is that it looks like a set of merge patches from branches representing each of the repositories.

(Maybe this is what you already intend, I'm just an anxious person)

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Sep 9, 2018

@jchesterpivotal

This comment has been minimized.

Copy link
Contributor

@jchesterpivotal jchesterpivotal commented Sep 10, 2018

Can bosh releases live in a subdirectory, or do they always have to sit at the root of a repository? If it can go in a subdirectory that suggests a relatively smooth rearrangement. git mv the current stuff into bosh-release/, then use the merges to bring the other repos in.

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Sep 10, 2018

@jchesterpivotal That'd work but I'd rather have a separate repo for the release anyway as I don't want our core code to appear married to any particular distribution method. I guess we could put them all in there, but then we'd lose GitHub issue fidelity, and I don't think we can do that for e.g. the Helm chart (since it has to be a fork of the charts repo for us to be able to PR changes back).

Current plan is to commit a rm -rf and merge the "one big merge" on top of that.

@vito

This comment has been minimized.

Copy link
Member Author

@vito vito commented Sep 26, 2018

We're now on "one big repo" and have new contributing docs written here:

https://github.com/concourse/concourse/blob/master/.github/CONTRIBUTING.md

Thanks to @tkellen for working with me on the contributing doc!

Here's a running tally of things done so far and things left to do, regarding the dev and contributor workflow:

  • replace dev/start with Docker Compose
    • make testflight able to run without needing direct access to workers
      • introduced Mock resource for this
      • this should also make testflight much faster to run, though I haven't done a comparison yet
    • optimize building/rebuilding/restarting
      • pre-bake modules and Go build cache into base image
      • pre-bake resource types into base image
      • auto-bump gdn and dumb-init (and any other deps)
    • auto propagate web assets to container so no restart/rebuild is needed
  • make web UI building feel more natural (Makefile -> yarn watch)
  • write a new CONTRIBUTING.md
  • get unit tests running against master in CI
  • get unit tests running for PRs
  • get testflight running against master in CI
  • get wats running for PRs
  • get testflight running for PRs
  • switch local baggageclaim driver to overlay (perhaps just switch this everywhere...) to prevent loopback leaks
  • move baggageclaim into its own pipeline
    • add jobs to publish major/minor/patch bumps
  • change acceptance tests to wait for ATC to be running prior to running tests
    • testflight (already does this!)
    • wats
    • watsjs
  • configure 'test' user for admin team and 'guest' user for non-admin test teams
@jchester

This comment has been minimized.

Copy link

@jchester jchester commented Oct 3, 2018

So I came back and took a look for my own hopes for this epic quest, which is to pull in various parts of Concourse as dependencies. It doesn't quite work, unfortunately.

My best guess, after resorting to pointless superstitious exercises like throwing runes, flipping tarot cards and going over what golang is pleased to call "documentation", is that the various subdirectories also need go.mod files.

Otherwise I get stuff like this:

pkg/check/check.go:8:2: unknown import path "github.com/concourse/concourse/atc": cannot find module providing package github.com/concourse/concourse/atc

Which prods me to try:

$ go get github.com/concourse/concourse/atc@3f6f1170443

go: finding github.com/concourse/concourse/atc 3f6f1170443
go: finding github.com/concourse/concourse 3f6f1170443
go get github.com/concourse/concourse/atc@3f6f1170443: missing github.com/concourse/concourse/atc/go.mod at revision 3f6f11704430

I also suspect, but can't prove without resorting to desperate measures like "effort", that go get doesn't understand the tag format Concourse uses for releases. If I say go get github.com/concourse/concourse@v4.2.1 it chokes.

Returning again to The Sacred Mysteries, I find the advice that "go get github.com/gorilla/mux obtains the latest version with a semver tag". It works for branches, it works for commit SHAs, but apparently just specifying a tag is rejected wholesale as the unclean work of beelzebub, seductive archfiend of convenient but missing features.

@vito vito closed this Jan 7, 2019
@vito vito added the accepted label Jan 7, 2019
@vito vito added this to the One Big Repo milestone Jan 7, 2019
@vito vito added this to Planned in One Big Repo via automation Jan 8, 2019
@vito vito moved this from Planned to Done in One Big Repo Jan 8, 2019
@vito vito added this to the v5.0.0 milestone Jan 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
One Big Repo
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.