Deprecation of Docker image for Insiders #2442
Comments
|
@fkorotkov – really interested in your view on this! |
squidfunk
changed the title
|
Thanks MD for your hard work and offering the Docker image. I've got no ideas how to make it more cost-efficient, but this is to say that I depend on this Docker image and could do it myself locally if guided. |
|
I build a wheel from Insiders locally and store the .whl file in my repository. Problem solved. At least if your repository is private like mine. I don't know what the Docker image is good for, but I apparently don't need it. |
|
I think there are couple of things worth to notice:
So the actual charges might be even smaller. IMO I think the "right" solution is for GH to provide an option to charge consumers. I've created a GitHub Community post with the idea. In the meantime, it will be inconvenient to build the image ourselves on every release, but it's not a critical inconvenience. If there will a simple instruction on how to automate the process so people don't accidentally make their images public, it will be a good option. |
|
Thank you guys!
That's correct. However, I cannot control where images are pulled from, so I have to assume "the worst", which is that they're pulled from outside GitHub. The charges might definitely be smaller, but it's still uncapped and out of my control. I don't plan to deprecate the Docker image tomorrow. I first wanted to get some feedback and there will definitely be a transition phase of 2-3 months from the moment of deprecation, where a warning will be issued when the image is invoked. I will provide a step-by-step guide how to build and push the image to your own private registry. In the end, it's as simple as forking the Insiders repository and creating a release on GitHub. This is the same process I'm following. The workflow contained in the Insiders repository is already configured to build and publish the image. As the fork is private, the Docker image will be too 😊 |
|
In my view, the easiest way to continue making use of docker images would
be to create/distribute a workflow that paid users will be able to add in
their private repo to build/push a docker image to their own private
ghcr.io/dockerhub/ecr registry.
To expand on this:
1. mkdocs-material provides a `image-build.yml` workflow file as part of
the insiders artifacts
2. a paid user creates a private repo where he adds just this workflow
3. in this private repo a user adds credentials for their container
registry of choice. the provided workflow may only include ghcr.io registry
as an example or all of them (dockerhub, ecr)
4. workflow can be as smart as checking if the available insiders release
has already been built and skip subsequent builds, or use a daily cron
job/manual_trigger for simplicity
4. at the end of the day each insider will have its own container image
built from the upstream insiders code and all the potential charges will be
solely on the users account (if they happen to be, which is unlikely if not
overused in CIs).
…On Mon, Mar 15, 2021 at 6:12 PM Fedor Korotkov ***@***.***> wrote:
I think there are couple of things worth to notice:
1. Pulls from GitHub Actions are not affecting the storage numbers.
2. I don't think total number of downloads represents the actual
downloads. I think it might be similar to Docker Hub where a cached pull is
also counted as a download.
So the actual charges might be even smaller.
IMO I think the "right" solution is for GH to provide an option to charge
consumers. I've created a GitHub Community post
<https://github.community/t/charge-consumers-for-packages-traffic/168076>
with the idea.
In the meantime, it will be inconvenient to build the image ourselves on
every release, but it's not a critical inconvenience. If there will a
simple instruction on how to automate the process so people don't
accidentally make their images public, it will be a good option.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#2442 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABLKV5OI7SIFEFBUFYU2LE3TDY5YNANCNFSM4ZFFN22Q>
.
|
|
Okay, so here's what's going to happen:
I believe that 2 months should be enough to transition to a self-hosted approach. Again, I'm sorry for any inconvenience this is causing you, but the reasons are stated in the original post and I don't see any way around this. I'm happy to collaborate on providing the best-possible documentation to host it yourself, it's very easy after all. |
|
Think the comment link in the bullet item 1 should be #2442 (comment), shouldn't it? |
|
Not quite, it won't be necessary to create a separate repository. You would just need to fork Insiders and issue a release on your fork – this will publish the Docker container to your private registry, and you will automatically be able to use it. |
|
My only concern here is that people will need to watch for new releases of insiders and manually pull the changes to the fork and issue a new release instead of just relying on the 'latest' tag here. Need to investigate how we can automate the "watching for new releases and mirroring them in our fork". Probably by having a cron action that syncs from the base repository and which build the image from a new tag rather then a release. But the issue there might be that an action can't trigger another action 🤔 |
|
Definitely a valid concern 👍 maybe we could automate this with a GitHub App, as this seems to be a general problem. Quick Googling yielded https://github.com/wei/pull (https://github.com/apps/pull). This could keep the fork in-sync to the upstream repository. We could then adjust the GitHub action in Insiders to automatically publish on a tagged branch, omitting the need to do a manual release. We could also scope this to forks, so the original repository is not impacted, or just comment it out, so the user would need to activate it. Thus, forking the repository and installing the pull app on it could be sufficient. |
|
I'm going to split out the Insiders overview and installation guides into a separate new section of the site. This section will also detail how to build the Docker image and publish it on your own registry. The current single-page solution worked very well, but I think we need to go into more detail to make it easier for users to get going. |
|
I prototyped a new workflow and think it is very straight forward:
The workflows have been adjusted so that forks will only invoke the |
|
Looks easy and doable. I've never done it before but can't wait to be a beta tester when requested! Happy to learn from you! 😎 |
|
I went ahead and moved up the deprecation date to today – it's done. The latest build of the Docker image contains a deprecation notice that links to this issue, to make users aware of the deprecation. On June 1, the deprecation will be effective, with the removal of the Docker image. Note that 2.4.0 is the last version of Insiders which is available as a Docker image. I wrote a new guide how to set up Insiders (pip, docker, git), with detailed instructions: Any feedback is appreciated! I think we should be all set, but of course, there might be some edges I haven't thought of. I tested the process with another account, which is how I wrote and tested the guide. |
squidfunk
changed the title
|
@squidfunk what do you think about renaming |
|
Hmm, I find that to be inconsistent with
Also, they seem to be pretty inconsistent about environment variable naming schemes, see: |
|
Reminder: the Docker image is going to be removed in 1 week. |
|
The Docker image was just removed. |
Update: the documentation explains how to self-host the Docker image with 5-10mins of effort.
In #2031, we paved the way to publish a Docker image that contains Material for MkDocs Insiders and is installable by opting into the GitHub Container Registry. Sadly, there are now two roadblocks ahead that make it uneconomical for me to further provide this service. This issue is meant as a discussion to find solutions, so please chip in, if you're a user of Insiders.
Transfer cost
As long as the GitHub Container Registry is in Beta, it's free to use for public and private repositories. However, when the Container Registry exits Beta stage, it's expected to have the same pricing as the GitHub Package Registry:
In the last months, we were able to squeeze the compressed size of the Docker image from 64 MB down to
2740 MB (see #2436 (comment)), which is an awesome achievement. However, still, the GB can add up quickly. If an upstream dependency like MkDocs chooses to add more dependencies, the image grows. If the image size stays at 40 MB, 1 GB equals roughly 25 downloads of the Insiders Docker image. This means that every download of the Insiders Docker image costs me 2ct. If somebody pulls an older image with a size of 64 MB, it's 3.2ct. This doesn't sound like much, and yes, of course Insiders are paying to get access to the repository. The problem is that I have no way to attribute downloads to certain users, I can only see the total number of downloads. This means I don't have any control about downloads (i.e. through rate-limiting), so a user could induce thousands of downloads and my bill would go up with no possibility to mitigate it, as I cannot know who's causing the traffic.For comparison, assuming 1 GB of data storage and 1.000 downloads (40 GB transfer out):
us-east-1): $ 3.54Of course, S3 is object storage, but this is just for a comparison of how expensive a private GitHub Package Registry is.
Programmatic access
The number of sponsors keeps growing at a steady pace, and up to now I have added and removed collaborators manually. Today, I started automating this process, but found out that this is not possible for the Container Registry, as there are no API endpoints that allow for doing that. I really hope that GitHub will add programmatic access at some point, but currently it's not possible.
Where to go from here
I understand that this might cause some of you trouble. I want to faciliate a discussion among the community, so we can maybe find a better solution that doesn't incur the problem with transfer cost getting out of control. Some possible solutions:
For 1., Amazon ECR would theoretically support more fine granular access control. This would however mean that we would need explicit account management, since we're now out of GitHub's realm. There's also still the problem for how to grant access to specific members of an organization, so I'm also investigating that. Maybe a GitHub App which is granted access on an organization / user repository could be an idea for better control over configuration and access management.
I'm open for ideas!
The text was updated successfully, but these errors were encountered: