Gated registry?

[crizCraig]

Wondering if it'd be possible to comply with Epic's licensing and grant access to images based on Epic Games GitHub membership. Or if you've talked with Epic about having them host a docker registry. From what I understand of the license terms, this would be very much in the same spirit as distributing via forks of the official repo. Perhaps just giving them access to a dockerhub would be the same or better for them, idk. Amazing project btw. So much better than everything else out there, including my own shell scripts :)

[adamrehn]

Hi Craig, I'm glad you've found the project useful.

This is something I've been pondering for a while now, since providing prebuilt images would be an extremely valuable resource for the community. In fact, the concluding paragraph of the first article I posted on my website regarding the Docker images described exactly this scenario:

With regards to the distribution of built images, Epic Games themselves are the only entity with the legal right to alter the current situation. I have no doubt that many Engine Licensees would be delighted if Epic Games were to provide automated builds of UE4 Docker images via an Epic-hosted Docker Registry. Such a registry could be integrated with the Epic Games account system to provide authentication and would significantly simplify the developer effort required for Engine Licensees to implement self-hosted CI pipelines.

I agree that an externally-hosted registry that restricts access based on GitHub/Epic-based authentication would be very much in keeping with the spirit of the Unreal Engine EULA. Unfortunately, the spirit of the law and the letter of the law (to borrow terminology from legal vernacular) are rather disjoint in this case, since the relevant section of the EULA is worded in extremely specific terms:

b. Distribution to other licensees - You may Distribute Engine Code (including as modified by you under the License) in Source Code or object code format, or any Content, to an Engine Licensee who has rights under its license to the same Version of the Engine Code or Content that you are Distributing.

Any public Distribution (i.e., intended for Engine Licensees generally) which includes Engine Tools (including as modified by you under the License) must take place either through the Marketplace (e.g., for distributing a Product’s modding tool or editor to end users) or through a fork of Epic’s GitHub UnrealEngine Network (e.g., for distributing source code).

I'm certainly not a lawyer, but my interpretation of that wording is that distribution of the Engine via a Docker Registry, even one to which access is strictly restricted to current Engine Licensees, would be a violation of the EULA. As such, the only viable option is for Epic Games to either host a container registry themselves and integrate it with the Epic Games account authentication system, or else create an organisation on Docker Hub that is linked to the EpicGames GitHub organisation. (Or to modify the wording of the EULA to relax how restrictive it is, of course!)

I've not reached out to Epic Games about this at the current point in time, but it's certainly a conversation I'd be interested to have with them if they're open to discussing it. I am planning on contacting them about another project of mine in the near future, so I'll endeavour to bring this up with them if the opportunity presents itself.

[crizCraig]

Hi Adam,

Thank you very much for your thoughtful reply. I've posted a topic to the Unreal Github forum to start a conversation with Epic. The post is in review as it's my first one :) - here's the content when it does go up:

Thanks to SonicIce and his amazing ue4-docker [1] project, packaging projects, continuous integration, server rendering and more inside Docker is now a breeze. It seems that the value of this work could be boosted greatly through an officially sanctioned Docker registry, saving devs all over the world countless hours in build time. Would using Git LFS storage of docker image tar balls [2] on GitHub forks of UnrealEngine be something Epic would recommend as a workaround for not having a Docker registry? Or is there some other way the great folks at Epic would recommend supporting this superpower of a use-case? Thanks in advance for your time answering this!

[1] https://adamrehn.com/docs/ue4-docker/read-these-first/introduction-to-ue4-docker
[2] https://docs.docker.com/engine/reference/commandline/save/#examples

[adamrehn]

Oops, I didn't realise I'd left my display name on the Unreal Forums in an inconsistent state compared to my other online presences. I've updated it from "SonicIce" to "Adam Rehn" now, so if you'd like to edit your post accordingly once it goes up, that should make things a little less confusing for everyone.

With regards to the use of Git LFS, the primary issue you'd likely run into is that GitHub places filesize limits on binary files stored using both LFS and GitHub releases. In the case of individual accounts the limits are low enough to preclude the possibility of storing 20GB or 30GB tarballs, but I have no idea what the limits are for enterprise customers or how those limits apply to the Unreal Engine repository and its forks.

[crizCraig]

Thanks Adam. I'll update your name when the post goes live. Interesting with regards to the size limits! I guess you could break the tarball into chunks within a release.

[Ludea]

Could you consider https://github.com/features/package-registry?ref=producthunt to store Docker images ?
I know there is UE4 license, you can not store images on free registry, but, you may deal with Epic repo access ?

[adamrehn]

Looking at the documentation for using Docker images with GitHub Package Registry, it looks like images are scoped to individual GitHub repos, and access is based on the permissions of the repo. In theory, that should mean that users could push images to the registry under the namespace of their own individual fork of the UnrealEngine repo and those images would then be accessible to everyone else in the Epic Games GitHub organisation, in exactly the same manner that the fork itself is accessible. I'm not a lawyer, but that would certainly seem to satisfy the wording "through a fork of Epic’s GitHub UnrealEngine Network".

I've just signed up for the Package Registry beta, so I'll test it out once I've gained access and confirm whether the permissions behaviour matches my interpretation of the documentation. If it does then we might finally have a legitimate means of sharing UE4 container images with other Engine Licensees!

[crizCraig]

This would be so awesome!!! It would also provide more transparency to Epic with regards to download numbers etc...

[TBBle]

Was there ever any movement on the GitHub Package Registry?

[adamrehn]

I signed up for the Beta waiting list, but I still haven't heard anything yet. I guess they must be rolling it out to participants in relatively small numbers at this stage.

[crizCraig]

Still waiting as well. I think I've signed up like four times which may hurt or help my cause :D

…

On Tue, Aug 6, 2019 at 6:07 PM Adam Rehn ***@***.***> wrote: I signed up for the Beta waiting list, but I still haven't heard anything yet. I guess they must be rolling it out to participants in relatively small numbers at this stage. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16?email_source=notifications&email_token=AABMH2K2ELRDJNGZQUAWW73QDIN3LA5CNFSM4GMY5MZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3W4IBY#issuecomment-518898695>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABMH2KTONDGUMNED24JCTLQDIN3LANCNFSM4GMY5MZA> .

[adamrehn]

Okay, I just got an email saying Package Registry will be releasing to general availability on November 13. Looking at the preview pricing, it's based on GBs of storage rather than image count, which is far from ideal for massive images containing the UE4 build tools, especially if individual users have to pay for images associated with their own forks of the Unreal Engine repo.

[crizCraig]

Yeah, that's unfortunate, why don't they have the same policy for releases? https://help.github.com/en/github/managing-large-files/distributing-large-binaries

[TBBle]

The pricing is public, and it charges on both storage and outgoing data transfer.

Since it's tied to the normal plans, does that mean it'd have to be Epic paying for it anyway? They're paying for all the private repo forks from their repo already.

[adamrehn]

@TBBle that's a good question, it's not clear how the pricing interacts with forks of organisation-owned repositories. That being said, even the 50GB storage option would be quite limiting when working with large Unreal Engine container images (especially the enormous Windows images) and it's not currently possible to delete image tags once they've been uploaded, which means the storage would fill up after pushing as few as two or three image tags and then remain full.

I'm still in the process of getting in touch with the Epic Games legal team to discuss image distribution options, so hopefully I'll be able to get some clarity from them that will allow us to move forward.

[dskvr]

@adamrehn I'm attempting to use GPR with publish-docker-images-to-gpr Github Action, but hitting some issues since the build context and dockerfiles are in a self-destructing temporary directory. These two deleted ue4-docker artifacts are required by the action.

Any tips?

[adamrehn]

@dskvr I'd advise against using GitHub Package Registry to store UE4 container images for the reasons I mentioned in my comment above, but if you do need access to the Dockerfiles then they're stored on the filesystem inside the ue4-docker Python package itself (e.g /usr/local/lib/python3.8/site-packages/ue4docker/dockerfiles). The raw Dockerfiles are copied to a temporary directory and then modified to add a message after each RUN directive that notifies users of filesystem layer commit overheads, but aside from that extra informational output there are no functional differences between the raw Dockerfiles from the Python package and the versions that are used by the ue4-docker build command.

It's worth noting that a future overhaul of ue4-docker will actually be structured around a two-phase process (generating Dockerfiles and then building the generated files) as discussed in #48, since I'm aware that the current way of doing things severely limits interoperability with external build systems.

[slonopotamus]

it's not currently possible to delete image tags once they've been uploaded, which means the storage would fill up after pushing as few as two or three image tags and then remain full.

It is now actually possible to delete images that were published to GitHub Package Registry, see answer at https://github.community/t/deleting-images-from-github-package-registry/2758/10

[adamrehn]

Container support has been spotted on the public roadmap for Unreal Engine 4.27 (https://portal.productboard.com/epicgames/1-unreal-engine-public-roadmap/c/320-ue-container-build-beta) and one of the dot points specifically mentions container distribution via GitHub:

Main features for 4.27:

• Provision of Dockerfiles that create container images for both Unreal Engine (Linux only) and Unreal Engine application (Windows and Linux) use cases.
• Distribution of prebuilt container images on Github Container Registry for convenience (Linux only).
• Documentation.

[adamrehn]

Almost three years after this issue was first opened, official pre-built container images for Unreal Engine 4.27 are now live on GitHub Container Registry: https://github.com/orgs/EpicGames/packages

You can find instructions for authenticating with GHCR and pulling a development image on the Containers Quick Start page of the Unreal Engine documentation.

Thankyou to everyone for your patience these past few years, and for your ongoing support of the project. With official adoption by Epic Games, the future looks bright for Unreal Engine containers.

[slonopotamus]

Yeah, but can we publish our own images in forks of UnrealEngine repo? =/ And who is going to pay for that?

[adamrehn]

@slonopotamus this is just for official images from Epic for now, but a mechanism for the community to share images is something that's being investigated.