Proposal: limit supported Unreal Engine versions to N latest releases

[adamrehn]

Problem overview

The current approach to Unreal Engine version support in ue4-docker is that we try to encompass as many releases as is feasible from a technical standpoint, and we only drop support for old releases when we make changes that break compatibility. For example, we dropped support for Unreal Engine 4.20 when we migrated to using Ubuntu 22.04 base images by default as part of #296, and then more recently in commit 84ca65c I introduced code that requires Unreal Engine 4.22 or newer. (Even then, the breaking code could easily be gated behind a check if we wanted to maintain support for Unreal Engine 4.21.)

Supporting a broad range of past Unreal Engine releases is helpful for users who cannot migrate to a newer version (e.g. due to working on a large project that has pinned the version of UE that it is built upon), but it also hinders the development of ue4-docker because it significantly increases the friction involved when making potentially breaking changes. Any change that might break old releases needs to be tested against all supported releases, and the test matrix has grown to an unwieldy size. In commit f249d1e I added a script to run image build tests for all supported releases, which is based on the testing I did as part of #296. Running this script in serial can take days, even on reasonably powerful hardware, and if the built images aren't automatically removed after each test then the disk space on most machines will be exhausted before completing all of the tests. Running the script in parallel reduces this time, but requires a sufficient number of machines or VMs, and that number only continues to grow as the list of supported releases grows. (Then multiply all of that by two if you're testing changes to both the Linux and Windows Dockerfiles.)

At the time of writing, ue4-docker supports the following Unreal Engine releases:

• 4.22, released in April of 2019
• 4.23, released in September of 2019
• 4.24, released in December of 2019
• 4.25, released in May of 2020
• 4.26, released in December of 2020
• 4.27, released in August of 2021
• 5.0, released in April of 2022
• 5.1, released in November of 2022
• 5.2, released in May of 2023
• 5.3, released in September of 2023

When Unreal Engine 5.4 is released (presumably sometime in Q2 of this year, based on the previous release dates), then ue4-docker will support a total of 11 releases that span a period of approximately 5 years. Although this is excellent for users running older UE releases, the testing burden of such a broad range is rapidly becoming impractical.

Proposed solution

To better balance the needs of users against the testing requirements for ue4-docker development, I propose that we introduce a policy of supporting a fixed number of Unreal Engine releases. Under this approach, we would drop support for the oldest version in the test matrix each time a new supported version of UE is released. This would provide the following benefits:

• A fixed upper bound for the size of the test matrix

• An easy way to reason about which versions of ue4-docker support which versions of the Unreal Engine. Rather than needing to examine the git history in order to determine the newest version of ue4-docker that supports a given UE release, users could simply look at the release notes to see when support for new UE versions was introduced, and calculate the supported range of each release from that. (Alternatively, we could maintain an explicit list of supported ranges in the documentation, and the proposed approach would make it easier for us to maintain that list.)

Discussion points

The key parameter for the proposed approach is of course the number of Unreal Engine releases to support. Since the transition to Unreal Engine 5, Epic seems to have settled into a cadence of two releases per year. If we assume this release cadence will continue, then the number of years covered by a given number of supported releases would look like so:

• 10 supported releases == 5 years (equivalent to the current test matrix)
• 8 supported releases == 4 years
• 6 supported releases == 3 years
• 4 supported releases == 2 years
• 2 supported releases == 1 year

My understanding is that it's fairly typical for projects (particularly games) to stick with the same release of the Unreal Engine for the entirety of their development cycle, and that development cycles in the order of 3-5 years are not uncommon. As such, I suspect that supporting only two or four UE releases may cause frustration for users who wish to adopt containers partway through the development of their project (and are unable to use the official images from Epic, due to using a custom engine fork or needing Windows images). At the same time though, supporting eight or ten UE releases results in little to no reduction in testing burden when compared to the current approach.

My initial instinct is to take the middle ground, and support 6 releases. Three years is a pretty reasonable timespan, and long-lived projects that decide to adopt containers after the three-year mark can always use an older version of ue4-docker to build their images. That being said, the same logic does arguably apply to 4 releases / 2 years, and the additional reduction in test matrix size would be nice, so I'm a bit torn.

Interested to hear your thoughts, @slonopotamus and @TBBle.

[slonopotamus]

Ideally, we should make the decision based on stats from users, but we don't collect any telemetry. 6 sounds fine, and we can always adjust if it happens to be too short.

[adamrehn]

Okay, I've updated the test matrix and README in commit 6a84001 to reflect a policy of supporting 6 releases.

I'll leave this issue open for the time being, and if users run into any concerns regarding the 6-release support policy then they can comment in this thread.

[gstorer]

Can I suggest that 4.27 (or 4.27-plus explicitly) is supported as long as Epic continue to support their 4.27-plus branch with SDK updates and fixes? https://github.com/EpicGames/UnrealEngine/tree/4.27-plus . 4.27-plus seems to be a sudo UE4 LTS branch.

[TBBle]

If we treat 4.27+ as UE4 LTS, it might make sense to also have a branch of this project that maintains 4.27+ support, since it shouldn't see many changes, and that will allow the main branch to follow UE evolution more easily by dropping 4.27+-specific support, bug-fixes, and workarounds, e.g., since we're still on 0.0.x versions, we could declare 0.1.x onwards does not support UE 4.27+, and keep making 4.27+ fixes in 0.0.X releases.

I guess it depends on how "LTS" 4.27+ really is; a quick glance suggests a fair bit of backporting is going on there, so there may not be that much value in branching ue4-docker compared to the admin overhead and extra effort needed by users dealing with both versions at once.