build instructions: docker is not alternative path #87
Comments
This was referenced Oct 3, 2020
bpkroth
added a commit
that referenced
this issue
Oct 13, 2020
Lots of build/pipeline cleanup with the goal of making and publishing standard Docker images the for Linux developers and the CI pipelines (#36). - Most implementation notes are in `.github/workflows/main.yml`, but here's a brief summary: - Nightly we build fresh docker images for Ubuntu {16.04, 18.04, 20.04}. They are only published if all of the usual build/test jobs pass (using the same logic as for PRs). That also helps prime the local caches for the following: - On each PR, we 1. (re)build the image from a local buildx cache that's keyed on the sha of the PR, and optionally will also pull from the latest "main" build (since it presumably worked). That is initially only stored in a local cache. Part of the goal here is to avoid the need to push unvetted docker images (e.g. tagged by sha) to the public registry and then have to clean them up later (which is not always even possible). 2. For each build/test job, we again "rebuild" the image, using that cache from step 1, so that it goes quickly. Then we load the image into the local runner's docker daemon and start an instance to run our tests. This reload is necessary because individual jobs are assigned to new runners so we can't reuse state between jobs. Jobs are organized into dependency graphs to account for this cache reuse. At the moment, all of the tests run on all versions of Ubuntu, which makes the CI pipeline a bit slow. - DONE: We could consider only running 16.04 and 20.04 in the CI pipelines and then running 18.04 in the nightly scheduled run or else just drop support for it entirely. 3. Once all of the tests pass, and the PR is merged, we execute a final job to update the published version of the job by retagging it and pushing it to various registries. The current registries are all Github services (for convenience to get started): - docker.pkg.github.com: this is nice because it supports the built in `GITHUB_TOKEN` secret so doesn't require any additional work for a forked repo to use for testing (they push to their own path though). However, it doesn't support anonymous pulls and is being deprecated by ... - ghcr.io: this the newly announced Github Container Registry service. It requires a special set of secrets to be assigned to the repo, so needs some additional work to make use of. Also, as of now, the Microsoft org doesn't seem to have it enabled, so for now I've set it to push to the Microsoft-CISL org. Additional work: - Reorg the Dockerfile to use multi-stage builds to allow skipping copying the source code into it so that we only distribute the build tools. We can also optionally build slimmer "python-only" images this way. In a future PR we'll take on the work to switch from straight pip to conda (#83) - TODO: Could optionally prime the `/src/MLOS` folder in the image so that one could `git pull` immediately to get the latest code if they wanted to without a local copy as a bind mount. - Added a `make docker-image` target to simplify repeated local docker image build invocations. - Switch to using tools (cmake, dotnet) installed in the system rather than fetched during build (#109) - These scripts are also nice reusable elements for users to install in their own environments if they wish. - DONE: Update the documentation to reference them instead of relying on the old "fetch" method. - DONE: There is work in ADO that provides a similar `install.python.sh` script to setup the `python3.7` install. Will port that over later as well. - Cleanup _most_ of the build system to match (I have not yet removed the old setup scripts, but they are no longer automatically called). - Added vscode integration support for developing inside the container (#105, #2) - Added support for running as a non-root user so that bind mounts of the repo source tree work without re-chowning files on the host system as root. This required a fair bit of fixups, but I think should cut down on some issues with students/others and makes the vscode integration easier as well. - Add nightly run scheduling of Python long haul tests (#113) - TODO: Fixup docker related documentation (#87)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Looking at the build docs it looks like there are three paths, docker, linux and windows:
https://microsoft.github.io/MLOS/documentation/02-Build/#docker
But docker is actually "just" how to install docker, so you still need to run the linux installation afterwards.
I think we should restructure the docs to make this more clear, or just say "now do the linux build" after the docker install.
The text was updated successfully, but these errors were encountered: