Add Julia devcontainer

[davidanthoff]

I don't understand the testing infrastructure, so I just left the test script from the template in here. Should I delete that, or do something else with it?

[Chuxel]

Generally we're trying to keep the postCreateCommand as something developers can use as needed. These files are dropped into an existing project. Is this something that every developer would want to do with source files that already exist?

[davidanthoff]

Yes, I think this makes sense for everyone. It essentially just makes sure all the packages that are specified as a dependency in the root Julia configuration file in the repo are actually installed on the machine and available right away.

I guess in theory we could also try to put this into a docker script file, but then we would lose the ability to use the pre existing julia image files and instead would end up using building a new image for every single repo, which seems a lot less nice from a performance point of view?

We've been doing a very similar thing in the existing Julia mybinder.org integration for a long time and it seems to be what users want.

[Chuxel]

Yeah, there's a bit of a balance between keeping it simple and getting people started that varies between ecosystems.

Generally we've been adding Dockerfiles to help people get started on adding their own contents to the image. (For example, this Dockerfile). That's not strictly required though - it depends on how likely someone is to want to do something like this.

One thing I did notice, however, is git is not present in the base Julia image. That's a pretty common dependency. There is a script that can be used that pulls in common dependencies including git. You are right that it can increase the time it takes to get the container up, however. You can default to not installing zsh and not upgrading packages to help there a bit. e.g.

# This Dockerfile adds a non-root user with sudo access. Update the “remoteUser” property in
# devcontainer.json to use it. More info: https://aka.ms/vscode-remote/containers/non-root-user.
ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=$USER_UID
# Options for common setup script
ARG INSTALL_ZSH="false"
ARG UPGRADE_PACKAGES="false"
ARG COMMON_SCRIPT_SOURCE="https://raw.githubusercontent.com/microsoft/vscode-dev-containers/master/script-library/common-debian.sh"
ARG COMMON_SCRIPT_SHA="dev-mode"
# Install needed packages and setup non-root user. Use a separate RUN statement to add your own dependencies.
RUN apt-get update \
    && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends curl ca-certificates 2>&1 \
    && curl -sSL  ${COMMON_SCRIPT_SOURCE} -o /tmp/common-setup.sh \
    && ([ "${COMMON_SCRIPT_SHA}" = "dev-mode" ] || (echo "${COMMON_SCRIPT_SHA} */tmp/common-setup.sh" | sha256sum -c -)) \
    && /bin/bash /tmp/common-setup.sh "${INSTALL_ZSH}" "${USERNAME}" "${USER_UID}" "${USER_GID}" "${UPGRADE_PACKAGES}" \
    && rm /tmp/common-setup.sh \
    && apt-get autoremove -y \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

The package upgrade is mainly something we have for scenarios where a downstream image is actually generated so security patches are picked up before its published.

[davidanthoff]

Ah, git is a good point! I'll try to figure out whether there is some pre-built image for julia that includes git, that would give us the fastest story, right? If not, I'll look into the custom dockerfile.

[davidanthoff]

Or would it make sense that we (the Julia extension for VS Code) publish our own julia-devcontainer image for this scenario? That way we could include anything we think is useful, but avoid the extra time to build a new docker image every time a user instantiates a new devcontainer?

I guess this is a scenario where microsoft/vscode-remote-release#3441 would also help: you (the codespace implementation) could actually cache the docker image that gets build by a particular devcontainer definition that is in the central repo and then reuse that docker image for any user instance that references it. mybinder.org has a pretty neat design where various docker images get cached at various points, there might be some good ideas there.

[Chuxel]

Publishing your own image makes a ton of sense since that gives you control over when updates need to happen to the image. There's a number of other examples of this in the repository - e.g. Puppet, Salesforce, Azure Functions.

The thing about microsoft/vscode-remote-release#3441 that is not covered by the image alone is the various devcontainer.json settings. In some cases these can be more involved, so we've considered the idea of a "inheritance" for it where you can start from a definition and then override, but that doesn't diminish the value of a published image.

[davidanthoff]

Ok, the custom image works great. I just need to figure out how we can update it regularly and make those kinds of aspects a bit more stable, and then I'll update this PR here. Thanks for all the help and guidance!

[Chuxel]

Re: Test infrastructure, there's a job that runs through and fires "test-project/test.sh" on a regular basis after spinning up the container. If the file has a non-zero exit code and we determine it's not related to an extension change, we'd raise an issue here and CC the owner of the definition that failed.

So, you can put anything in there you think is needed to smoke test the Julia image in this context.

That help?

[davidanthoff]

So, you can put anything in there you think is needed to smoke test the Julia image in this context.

Alright, I just added a test that runs the Julia binary with the --version flag, that should be enough to check whether that worked.

Let me know if you think I should change anything else with this PR, I think from my end this is ready.

[Chuxel]

Nice! Looks like we posted at the same time. One comment above: #450 (comment)

[ThomasHagebols]

@davidanthoff Nice work! Really cool feature 😄

I noticed that some changes were requested by @Chuxel. I started on the requested changes here. It is still a work in progress. Once I'm satisfied what would be the best way to contribute? Should I ask @davidanthoff to merge my changes or should I make a PR to microsoft/vscode-dev-containers directly?

[davidanthoff]

@ThomasHagebols very cool the progress you made there!

The thing where I got stuck is that I thought it would probably be best if we hosted our own Julia VS Code docker image somewhere instead of adding a full Dockerfile as part of the template to every user repo. The main benefit would be that if we want to make a change to the image, we could do that centrally and even repos of users that added the template once already would benefit from that. And then I got kind of stuck how we might host the image, how we would handle the versioning story of the image etc... but I still think it would be better if the template we add had just something like "image": "julia-vscode:latest" in it, rather than a full Dockerfile.

I think we should probably host these images on the just announced Github Container Registry? I'll try to figure out how that actually works, and then I guess we would use the Dockerfile that you created to create the images that we want to upload to there?

[ThomasHagebols]

Agree, having "image": "julia-vscode:latest" would be very neat 😄 The suggested changes were intended as an temporary solution until the questions around maintaining a custom image have been solved.

Where would you suggest to host the the Dockerfile / image? https://github.com/julia-vscode/julia-vscode? Let me know if I can contribute in any way

[davidanthoff]

@ThomasHagebols, yes would be great if you could help with this! I've set up all the infrastructure to host our own images etc. If you could update the Dockerfile that we're using with the changes you've already made, it would be awesome!

The Dockerfile is hosted at https://github.com/julia-vscode/julia-devcontainer. You could just open a PR against that. The JULIA_VERSION should stay with that name, because that then integrates with the Github Action that pushes the images.

The images are then hosted at https://github.com/orgs/julia-vscode/packages/container/julia-devcontainer/. I still need to fix which of the versions that we host it considers the "latest", other than that it should all work.

https://github.com/davidanthoff/StringBuilders.jl/blob/master/.devcontainer/devcontainer.json has an example of how this can be used. Essentially it is now "image": "ghcr.io/julia-vscode/julia-devcontainer", and then one can add a Julia version specifier at the end of that, for example "image": "ghcr.io/julia-vscode/julia-devcontainer:1.5" etc.

Once we have the Dockerfile updated over there, I'll update this PR here to use our new images.

[ThomasHagebols]

Cool, I can pick it up end of week. I will update the Dockerfile and update the accompanying readme

[davidanthoff]

So, I think this is ready to be merged, thanks to @ThomasHagebols' help!

The repo that builds our docker images is at https://github.com/julia-vscode/julia-devcontainer, and the images themselves are published at https://github.com/orgs/julia-vscode/packages/container/package/julia-devcontainer.

Once things are merged here, it would automatically show up in some new release of VS Code, right? Or the remote extension?

[Chuxel]

LGTM! One note is that we're discussing switching the default user from "root" to the "vscode" user, but given that's already in the image, I can add that to the PR I'll submit on the topic. I'll also make a couple of tweaks to the readme and add a comment pointing to the source for the image just so people have it.

That said, yes, the next release we do out of this repository will have the definition in it.

[davidanthoff]

Fantastic, just discovered it in the shipping build :)

[jlperla]

@davidanthoff Would you be willing to consider a devcontainer with conda + jupyter + the python datascience extension merged in as well?

[davidanthoff]

Ok, now that I have tried it, I think it is not yet ideal: it now adds a README.md and a .npmignore file to the repo if a user adds the template. I don't think we want that, right? I thought the README.md would be for information purposes here, but it certainly doesn't seem like the kind of README a user would want to add to their repo just because they add the devcontainer info. It even prompts the user to overwrite any existing README, which I think it really shouldn't do. Same with the .npmignore, I don't think there is any reason to add that, right? Is the right way to address this to just remove those two files from the repo here? Or can we somehow mark them as "these files aren't part of the stuff you should drop into a user's folder"?

Another, unrelated question: @pfitzseb had a better idea how we could a) simplify the postCreateCommand command and b) allow us to update it centrally. We could embed a Julia postcreatecommand.jl script in the custom docker image that we are using now, and we could then change the postCreateCommand to simply run julia path_in_docker_image_to_postcreatecommand.jl. I think that would make the template look much more streamlined, and it would allow us to update the post create command in a much better way. @Chuxel, would that make sense to you?

And another question: I've seen definition-manifest.json files in a couple of places now, is there some documentation for those somewhere? It seems that would give us the option to add some choices? We could for example use that to allow the user to select which Julia version they want to use?

@jlperla I think that should probably be a different devcontainer, I'd like to keep the default Julia one simple and lean. But having a "all things datascience" devcontainer template in general seems like a good idea.

[jlperla]

@jlperla I think that should probably be a different devcontainer, I'd like to keep the default Julia one simple and lean. But having a "all things datascience" devcontainer template in general seems like a good idea.

100% agree. After this gets solid we can see if merging in the minimal jupyter stuff necessary into a julia-jupyter or something like that.

[dlfivefifty]

Are there instructions on how to use this?