Repository navigation
Deprecation of Docker image for Insiders #2442
Description
Activity
- pinned this issue
on Mar 14, 2021 @fkorotkov – really interested in your view on this!
- changed the title
[-]Intent to deprecate: Docker image for Material for MkDocs Insiders[/-][+]Intent to deprecate: Docker image for Insiders[/+]on Mar 14, 2021 Thanks MD for your hard work and offering the Docker image. I've got no ideas how to make it more cost-efficient, but this is to say that I depend on this Docker image and could do it myself locally if guided.
Reacted by Martin DonathI build a wheel from Insiders locally and store the .whl file in my repository. Problem solved. At least if your repository is private like mine. I don't know what the Docker image is good for, but I apparently don't need it.
Reacted by Martin DonathI think there are couple of things worth to notice:
- Pulls from GitHub Actions are not affecting the storage numbers.
- I don't think total number of downloads represents the actual downloads. I think it might be similar to Docker Hub where a cached pull is also counted as a download.
So the actual charges might be even smaller.
IMO I think the "right" solution is for GH to provide an option to charge consumers. I've created a GitHub Community post with the idea.
In the meantime, it will be inconvenient to build the image ourselves on every release, but it's not a critical inconvenience. If there will a simple instruction on how to automate the process so people don't accidentally make their images public, it will be a good option.
Thank you guys!
Pulls from GitHub Actions are not affecting the storage numbers.
That's correct. However, I cannot control where images are pulled from, so I have to assume "the worst", which is that they're pulled from outside GitHub. The charges might definitely be smaller, but it's still uncapped and out of my control.
I don't plan to deprecate the Docker image tomorrow. I first wanted to get some feedback and there will definitely be a transition phase of 2-3 months from the moment of deprecation, where a warning will be issued when the image is invoked. I will provide a step-by-step guide how to build and push the image to your own private registry. In the end, it's as simple as forking the Insiders repository and creating a release on GitHub. This is the same process I'm following. The workflow contained in the Insiders repository is already configured to build and publish the image. As the fork is private, the Docker image will be too 😊
Reacted by Fedor Korotkov- In my view, the easiest way to continue making use of docker images would be to create/distribute a workflow that paid users will be able to add in their private repo to build/push a docker image to their own private ghcr.io/dockerhub/ecr registry. To expand on this: 1. mkdocs-material provides a `image-build.yml` workflow file as part of the insiders artifacts 2. a paid user creates a private repo where he adds just this workflow 3. in this private repo a user adds credentials for their container registry of choice. the provided workflow may only include ghcr.io registry as an example or all of them (dockerhub, ecr) 4. workflow can be as smart as checking if the available insiders release has already been built and skip subsequent builds, or use a daily cron job/manual_trigger for simplicity 4. at the end of the day each insider will have its own container image built from the upstream insiders code and all the potential charges will be solely on the users account (if they happen to be, which is unlikely if not overused in CIs).…On Mon, Mar 15, 2021 at 6:12 PM Fedor Korotkov ***@***.***> wrote: I think there are couple of things worth to notice: 1. Pulls from GitHub Actions are not affecting the storage numbers. 2. I don't think total number of downloads represents the actual downloads. I think it might be similar to Docker Hub where a cached pull is also counted as a download. So the actual charges might be even smaller. IMO I think the "right" solution is for GH to provide an option to charge consumers. I've created a GitHub Community post <https://github.community/t/charge-consumers-for-packages-traffic/168076> with the idea. In the meantime, it will be inconvenient to build the image ourselves on every release, but it's not a critical inconvenience. If there will a simple instruction on how to automate the process so people don't accidentally make their images public, it will be a good option. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2442 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLKV5OI7SIFEFBUFYU2LE3TDY5YNANCNFSM4ZFFN22Q> .Reacted by Martin Donath
Okay, so here's what's going to happen:
- The Insiders Docker image will be deprecated on April 1. The documentation will be adjusted to explain why the Docker image was deprecated and how to build the Insiders Docker image and host it on your own registry (basically as described in Deprecation of Docker image for Insiders #2442 (comment)). All releases after April 1 will not be accompanied by a Docker image.
- Inside the Docker image, the entrypoint will be wrapped with a shell script which will print a big warning on invocation that the image will be deprecated, linking to the corresponding section of the documentation.
- All sponsors joining after April 1 will not be granted access to the Insiders Docker image – it wouldn't make sense, because we're in the deprecation phase and they would need to switch back after a few weeks.
- After a transition phase of 2 months, the Insiders Docker image will be removed on June 1.
I believe that 2 months should be enough to transition to a self-hosted approach. Again, I'm sorry for any inconvenience this is causing you, but the reasons are stated in the original post and I don't see any way around this. I'm happy to collaborate on providing the best-possible documentation to host it yourself, it's very easy after all.
Think the comment link in the bullet item 1 should be #2442 (comment), shouldn't it?
Not quite, it won't be necessary to create a separate repository. You would just need to fork Insiders and issue a release on your fork – this will publish the Docker container to your private registry, and you will automatically be able to use it.
My only concern here is that people will need to watch for new releases of insiders and manually pull the changes to the fork and issue a new release instead of just relying on the 'latest' tag here.
Need to investigate how we can automate the "watching for new releases and mirroring them in our fork". Probably by having a cron action that syncs from the base repository and which build the image from a new tag rather then a release. But the issue there might be that an action can't trigger another action 🤔
Definitely a valid concern 👍 maybe we could automate this with a GitHub App, as this seems to be a general problem. Quick Googling yielded https://github.com/wei/pull (https://github.com/apps/pull). This could keep the fork in-sync to the upstream repository. We could then adjust the GitHub action in Insiders to automatically publish on a tagged branch, omitting the need to do a manual release. We could also scope this to forks, so the original repository is not impacted, or just comment it out, so the user would need to activate it. Thus, forking the repository and installing the pull app on it could be sufficient.
I'm going to split out the Insiders overview and installation guides into a separate new section of the site. This section will also detail how to build the Docker image and publish it on your own registry. The current single-page solution worked very well, but I think we need to go into more detail to make it easier for users to get going.
I prototyped a new workflow and think it is very straight forward:
- Fork the Insiders repository to your own or organization account
- Enable GitHub Actions on the new repository
- Create a personal access token with the
write:packagesscope - Add this token to the secrets of the fork under
GHCR_TOKEN - Optional: install GitHub Pull App to automatically stay in-sync with upstream
The workflows have been adjusted so that forks will only invoke the
publishworkflow on new tags that match the#.#.#scheme. Thebuildanddocumentationworkflows are essentially no-ops, and can be deleted (but won't run anyway, so no need). When a new release is issued, a new tag is created, the tag is synced (manually or via the Pull App), and the release is automatically published on the registry of the user or organization account. I think that's pretty much it, and it should be achievable in 5-10 minutes of setup 😊 I'm currently in the process of adding this to the Insiders documentation, but I wanted to share it here if users would already want to try it out.Reacted by Jacek LaskowskiLooks easy and doable. I've never done it before but can't wait to be a beta tester when requested! Happy to learn from you! 😎
I went ahead and moved up the deprecation date to today – it's done. The latest build of the Docker image contains a deprecation notice that links to this issue, to make users aware of the deprecation. On June 1, the deprecation will be effective, with the removal of the Docker image. Note that 2.4.0 is the last version of Insiders which is available as a Docker image.
I wrote a new guide how to set up Insiders (pip, docker, git), with detailed instructions:
https://squidfunk.github.io/mkdocs-material/insiders/getting-started/#with-dockerAny feedback is appreciated! I think we should be all set, but of course, there might be some edges I haven't thought of. I tested the process with another account, which is how I wrote and tested the guide.
- changed the title
[-]Intent to deprecate: Docker image for Insiders[/-][+]Deprecation of Docker image for Insiders[/+]on Mar 21, 2021 - unpinned this issue
on Mar 21, 2021 @squidfunk what do you think about renaming
GHCR_TOKENtoCR_PATsince this is how the personal access token is referred in GitHub's documentation and some people can already have it configured on organization level. What do you think?Hmm, I find that to be inconsistent with
GH_TOKENwhich is used in many places around the web (and in our own docs). If it trips up users, we can surely rename it, but our docs refer toGH_TOKEN, and I thought thatGHCR_TOKENpretty much follows from it.GH_TOKEN– GitHub tokenGHCR_TOKEN– GitHub Container Registry token
Also, they seem to be pretty inconsistent about environment variable naming schemes, see:
https://docs.github.com/en/packages/guides/migrating-to-github-container-registry-for-docker-images#migrating-a-docker-image-using-the-docker-cliReminder: the Docker image is going to be removed in 1 week.
The Docker image was just removed.
Update: the documentation explains how to self-host the Docker image with 5-10mins of effort.
In #2031, we paved the way to publish a Docker image that contains Material for MkDocs Insiders and is installable by opting into the GitHub Container Registry. Sadly, there are now two roadblocks ahead that make it uneconomical for me to further provide this service. This issue is meant as a discussion to find solutions, so please chip in, if you're a user of Insiders.
Transfer cost
As long as the GitHub Container Registry is in Beta, it's free to use for public and private repositories. However, when the Container Registry exits Beta stage, it's expected to have the same pricing as the GitHub Package Registry:
In the last months, we were able to squeeze the compressed size of the Docker image from 64 MB down to
2740 MB (see #2436 (comment)), which is an awesome achievement. However, still, the GB can add up quickly. If an upstream dependency like MkDocs chooses to add more dependencies, the image grows. If the image size stays at 40 MB, 1 GB equals roughly 25 downloads of the Insiders Docker image. This means that every download of the Insiders Docker image costs me 2ct. If somebody pulls an older image with a size of 64 MB, it's 3.2ct. This doesn't sound like much, and yes, of course Insiders are paying to get access to the repository. The problem is that I have no way to attribute downloads to certain users, I can only see the total number of downloads. This means I don't have any control about downloads (i.e. through rate-limiting), so a user could induce thousands of downloads and my bill would go up with no possibility to mitigate it, as I cannot know who's causing the traffic.For comparison, assuming 1 GB of data storage and 1.000 downloads (40 GB transfer out):
us-east-1): $ 3.54Of course, S3 is object storage, but this is just for a comparison of how expensive a private GitHub Package Registry is.
Programmatic access
The number of sponsors keeps growing at a steady pace, and up to now I have added and removed collaborators manually. Today, I started automating this process, but found out that this is not possible for the Container Registry, as there are no API endpoints that allow for doing that. I really hope that GitHub will add programmatic access at some point, but currently it's not possible.
Where to go from here
I understand that this might cause some of you trouble. I want to faciliate a discussion among the community, so we can maybe find a better solution that doesn't incur the problem with transfer cost getting out of control. Some possible solutions:
For 1., Amazon ECR would theoretically support more fine granular access control. This would however mean that we would need explicit account management, since we're now out of GitHub's realm. There's also still the problem for how to grant access to specific members of an organization, so I'm also investigating that. Maybe a GitHub App which is granted access on an organization / user repository could be an idea for better control over configuration and access management.
I'm open for ideas!