Completely Refactor the SAR Release System/Schedule

[JaioCG]

I feel it's not an unpopular opinion to say that the current SAR release system has some major flaws to it, with the current "stable" release (1.12.8) broken in multiple ways (quitting being broken, solocoop being broken, etc.), and any time the broken-ness of the system its mentioned, it's always quickly shut down with a "Do things yourself, bucko", which at least to me is pretty Royally toxic. Hopefully making a full issue about it can put some eyes onto making changes not just to how SAR updates are made, but to the entire release system in gemeral.

Important

This is just my opinion of how things currently are in my mind, and how they can be changed, not the direct route that things really are or how they should go. Obviously, suggestions and corrections are welcome in the form of replies.

The Current Release System

"Stable" Releases vs. Pre-Releases

SAR in its current state is broken up into two main versions, the main "stable" release, and pre-releases. As of writing this, the current "stable" release is 1.12.8, and the latest pre-release is 1.13.0-pre4. "Stable" releases are relatively rare, happening only say once per year, while pre-releases are treated like main releases, only with the thought that if a feature is broken or illegitimate, then runs using the pre-release can be banned.

With pre-releases effectively functioning as the only kind of release, large feature-set releases (like 1.13.0-pre4) and single-line literal crash fixes (like 1.12.8-pre6) are treated the exact same. This causes issues when there is a crash or any other game breaking issue that needs solving, it could be pushed off for release just to add in some extra features (like the upcoming 1.13.0-pre5 fixing the +leaderboard issue).

"Stable" releases (I put "stable" in quotes because 1.12.8 is very much not stable lol) really only happen when SR.C and CM admins bug SAR devs enough to get a full release, due to the rules on legal SAR versions. Because of this, the "stable" 1.12.8 release only really released with some extra TAS features and fixes, while the pre-releases added a whole lot more (that wasn't adjusted for the full release) and fixed multiple crashes. Obviously yes, I know pre-releases are meant to actually implement the features in their initial states to be fixed/updated in feature pre-releases, but this current system puts no weight on full releases, making them feel like just a number changing for the same of game admins. I suspect 1.13.0's full release will be similar, with the pre-releases having features like viewmodel shadow removing, Portal Reloaded co-op timing, new conds, co-op ghosts support, proper mod support on Linux, skipping workshop maps, multiple mtrigger changes, and more being fully fleshed out and finished, while the actual full release, or even any future pre-releases won't get much changes of what's before, essentially making those versions stable compared to the new.

A little side note about the "forbidden gamemode" or whatever. I know cmm is probably planning to go into beta somewhere around 1.13.0 (which I think was one of the reasons the minor value in the version was changed to 13), but at this current rate, a major feature addition like cmm won't feel any different to an update that only adds tiny changes, as they're treated identically as pre-releases.

Wait... How do Releases Work, Anyway?

The versioning of SAR is based on semantic versioning (but not as fully-set), following a MAJOR.MINOR.PATCH system. But currently, only the "PATCH" and extended pre-release version numbers are used, with MAJOR being stuck at 1 when there have definitely been updates that qualify for major version changes (speedrun timer rewrite, config+, overlay system rewrite for text/ghosts, etc.), and MINOR being stuck at 12 for multiple years, which is currently associated with an "era" of SAR's development, not based on new smaller-scale features and commands.

When it comes to timing, full releases only happen once in a while (the period between 1.12.7 and 1.12.8's release was approximately 1 year and 2 months), and pre-releases can come very sparatically, some in quick succession while others can have months between them. I think the sporadicness is part of the reason why quick fix updates are delayed to add in more features, as there may not be a chance for those new features to be released in any quick time window, and may sit in PR hell for months (I'm looking at you, Reloaded co-op timing). Obviously this is not ideal, but if every update is treated as a big feature-set update, then there may not be any hurry to do a release when the feature list is small, even if it is. As of right now, NeKz is really the only one doing releases, and with the sporadic nature of a busy schedule, there may be days between any communication updates between devs or runners testing out fixes for releases, which I believe is also hurting scheduling. A recent example of this being the +leaderboard changes in 1.13.0-pre4 being quickly fixed by AMJ, but without release permissions, being stuck in limbo while runners are being told to "just downgrade".

New Release System Proposition

I think the crux of the issue is that feature-set releases are treated identically to quick patches, both being labled as pre-releases, which can be seen with some of the updates I named above. Instead of actually using the MINOR.PATCH system that semver implements, or at least loosely trying to be similar, everything is dropped into a pre-release until SR.C or CM admins complain enough for a full release. I'm feeling safe with proposing a change here to fit more into standard versioning, hoping changes can be laid out easily.

Woah! Versioning That Makes Sense!

Start simply with releasing the current 1.13.0 as 2.0.0. I don't really have a reasoning for this, but it feels justified enough, and makes the rest of this new system work way easier. From there, MINOR versions can be used for feature-set releases, like 1.13.0-pre4, while patches are saved for neccesary quick fixes, like 1.12.8-pre6. These changes remove any stress of having to pack more features into what should be a quick bug-fixing release, as well as leaving pre-releases open to their original purpose.

In my mind, the correct use for a pre-release would be only adding/releasing an in-beta feature, say cmm, letting players demo and test out the feature before properly merging it into a full release. This gives players the opportunity to test one specific feature, and the ability to fall back into a main release when not currently playing. This also justifies the "if X is illegitimate then we can ban the runs" policy, as these new pre-releases can be much more experimental compared to how they are now.

Changing the versioning methods would also make the lives of leaderboard admins (hopefully) easier, not having to deal with multiple different versions of the plugin, can just support the one current "main" version, and maybe a pre-release if one is available at the time. I say "if one is available" assuming that pre-releases will only be used for in-dev features, then bento a full release. essentially removed once the feature is merged i

Commonly Done Releases?

As well, this should fix any issues with release timing. If a release isn't expected to include many features, then it can get released as soon as possible, especially if more people have permissions to create releases. Sure, MAJOR versions should be way between one another, only being for major code rewrites or additions, and MINOR versions probably shouldn't be released daily. But to avoid issues where features or neccesary fixes are stuck in PR's or just merged without release, those things should be done as quickly as possible.

Ending Statements

I understand that everyone working on SAR, srconfigs, rules, mtriggers, and every other repo related to P2SR is doing it on their own volunteer schedule, which is, and should be, in the background of more important things (school, work, health). But sometimes I do really feel like even though work is being done, it's not being done in the places that people are wanting to see the most, or in the places that actually matter. I don't expect releases, features, etc. to be done on the daily, I think the current development speed and feature-addition speed of P2SR tools is perfectly fine where it is (even if I think some features should be focused on more than others), but it just feels like releases are slow, and only happen when absolutely 100% neccesary, and only if enough features are added.

Sorry that this issue is so long, being a massive information, suggestion, and complain dump, but this is just one of the issues I have with P2SR tools development that people just ignore, and that I feel needs attention brought to. Especially sorry if it does feel like I'm just complaining here, but it's just infuriating knowing that the people that do have power to change things, or at least the most vocal in the community, shut things down instantly, or try to hide/fake-delay new things existing when the community is curious and jsut wants some- hell, any- communication.

I don't think I'll make any more of these for now as this one was pretty tiring to write, trying to mix information I see written out on Discord with my own opinions, and suggestions for changes that won't piss too many people off. I really hope something about releases can change though, as I know it's one of many users' main issues with SAR, and P2SR development as a whole.

[mlugg]

Hi, I'm coming out of my cave to give my opinion here. It should not carry as much weight as it would have in the past -- the decision here is ultimately up to AMJ and NeKz. Also, my responses here will probably be out-of-order and generally a bit all over the place -- sorry about that.

The versioning of SAR is based on semantic versioning

This is not, and never has been, true. If SAR tried to follow semver, almost every release would be major, and we'd probably have hit 50.0.0 by now: SAR has enough quirks that most non-trivial releases have breaking changes of some kind. An a.b.c format does not equal semver. Slightly weird analogy, but our versioning system is more akin to something like Minecraft -- it's more like 1.major.minor, but major and minor are based off of... vibes, to be honest.

This system, I will admit, is weird and ill-defined. However, that's... really not a problem? Semver is useful and important for software where backwards compatibility is important. For instance, a hypothetical Wormhole should follow semver (or a similar system) because mods rely on APIs exposed by specific Wormhole versions. SAR, on the other hand, has no such constraints, so going with a more "intuitive" system where the major version increments upon huge rehauls or changes of ownership, and minor increments on other releases, works pretty well IME.

Obviously yes, I know pre-releases are meant to actually implement the features in their initial states to be fixed/updated in feature pre-releases, but this current system puts no weight on full releases, making them feel like just a number changing for the same of game admins.

If you look at any piece of software with any kind of pre-release system, this will be, by design, how it works. Ideally, there would be zero code changes between the final prerelease in a release cycle and the following full release. Full releases are intended to be as battle-tested as possible, and come once bugs in a version -- particularly around important features like timing -- are mostly eliminated and the rate of new bug reports has fallen to near-enough zero.

I fail to see how this structure is a bad thing. The intention, as you seem to understand, is that users can use pre-releases at their own risk as needed; changes may be drastic or frequent, and bugs are to an extent expected. By the time a full release rolls around, the rate of changes should be effectively zero to guarantee stability and smooth upgrades for those tracking full releases.

When it comes to timing, full releases only happen once in a while [...] and pre-releases can come very sparatically (sic) [...]

The release schedule follows the rate of development. Some years back, when I was very active, I would work on SAR near enough every day of my life for months on end. This meant the rate of pre-releases was naturally much higher, because there was more active development, and hence more changes to dogfood. Nowadays, the rate of development has slowed. This is in large part because the current active maintainers simply do not have the time I did during COVID and the start of my degree: the amount of time I put into SAR was pretty close to a full-time job a lot of the time. I do not say this to boast about the effort I put into this project (I'm well aware it probably sounds like that), but rather to point out that it is unrealistic to expect SAR to maintain anywhere near that level of progress. So, it is to be expected that releases are much more gradual now than they once were.

As you later acknowledge, the people working on SAR are doing so on a volunteer basis, and for most of them, speedrunning is not a particularly major part of their life. Statements like "there may be days between any communication updates between devs", to me, imply an expectation of time dedication that is simply not realistic. Try comparing the release schedule of SAR to that of sister projects like SourcePauseTool, which will generally have one release every 4 months, despite having several more active developers (as you pointed out, the current SAR team is basically AMJ and NeKz).

Changing the versioning methods would also make the lives of leaderboard admins (hopefully) easier [...]

The current system exists as a reasonable safeguard against the high chance of a bug affecting timing or otherwise having a major impact on the game being introduced. A release schedule akin to what you have proposed would essentially just promote the majority of what are currently pre-releases to full releases. This is not easier for admins. There would be just as many SAR versions floating around; the difference is that subtle timing or gameplay issues are more likely to be introduced into a full release, and these become very problematic to deal with. You either have to allow a bunch of bad times, or roll back virtually every time from a given interval despite the runners in question following all stated rules. This sucks for all parties involved.

But sometimes I do really feel like even though work is being done, it's not being done in the places that people are wanting to see the most, or in the places that actually matter.

Broadly speaking, contributors to SAR are giving their time because they enjoy it: there are features they want to introduce for themselves or for others, and they are willing to put time into those. In my opinion, no member of the community who is not putting their money where their mouth is and actively contributing to SAR has any authority to dictate the direction of the project's development -- even if admins or the community at large broadly want a certain feature.

---

There is one area I will agree with you on: bugfixes, particularly those affecting previously-functional components of SAR, ought to result in pre-releases as soon as possible. Further, if the latest full release of SAR has regressed due to game updates, that is a good reason to get a full release out as soon as possible. If I had the time and energy to be involved in active SAR development today, I would want to ensure this is the case. However, if the current maintainers disagree, that is their prerogative. I am about to grant @ThisAMJ full privileges in the @p2sr organisation (I actually thought I'd already given him release powers here, so apologies if I've been indirectly responsible for any frustration there) so he will have the power to make releases as he sees fit.

[ThisAMJ]

Just a quick note, I have had the power to release (I believe) as you say (here), but I'm a very non-committal and unconfident person, and I wasn't told if there was any release workflow to follow other than just "click Draft a new release," so I've left that to the qualified NeKz until now.

I am in agreement that a major release is necessary soon to address the regression, but I'm not confident some of the newer features are stable enough yet. Your judgement may vary. (especially the CM leaderboard handling in coop)

Because the game itself is so buggy at times, I believe people often credit issues with SAR to the base game and don't report SAR bugs as often as they should, GitHub issue creation notwithstanding.

[NeKzor]

I wanted to propose a new release schedule when I got back into SAR last year but there was almost no activity, except when there was activity we made a release. Now it makes more sense to introduce one if you look at the commit activity for the last few weeks. How about minor/patch releases every two weeks? There can be many patch releases per week. Of course this still only works if there is anything to release.

I honestly think we should remove pre-releases entirely and the philosophy that comes with it. Every new commit that lands in the master branch and that is not part of a release is basically a pre-release. If people want to try out the latest changes they should be able to use the update command without waiting for any release. I'm not exactly sure how this would work but for future references I will call this a "canary release".

A recent example of this being the +leaderboard changes in 1.13.0-pre4 being quickly fixed by AMJ, but without release permissions, being stuck in limbo while runners are being told to "just downgrade".

When there is a regression then people should try to find a workaround first before doing a downgrade. For example the +leaderboard change can be fixed with alias ... "" that disables it. A downgrade should usually not happen but it can happen. To make it easier the update command should allow an optional version argument. This will also be useful for finding regressions in general.

But currently, only the "PATCH" and extended pre-release version numbers are used, with MAJOR being stuck at 1 when there have definitely been updates that qualify for major version changes (speedrun timer rewrite, config+, overlay system rewrite for text/ghosts, etc.)

I totally agree that a major releases should happen at some point since many things have changed over the past years that made SAR significantly better. For future releases the only difference between major and minor is the planning behind it.

Sorry that this issue is so long, being a massive information, suggestion, and complain dump, but this is just one of the issues I have with P2SR tools development that people just ignore, and that I feel needs attention brought to. Especially sorry if it does feel like I'm just complaining here, but it's just infuriating knowing that the people that do have power to change things, or at least the most vocal in the community, shut things down instantly, or try to hide/fake-delay new things existing when the community is curious and jsut wants some- hell, any- communication.

I don't think I'll make any more of these for now as this one was pretty tiring to write, trying to mix information I see written out on Discord with my own opinions, and suggestions for changes that won't piss too many people off. I really hope something about releases can change though, as I know it's one of many users' main issues with SAR, and P2SR development as a whole.

There is absolutely no need to apologize for anything lol. This is a valid issue and you spoke up. I hope to see more if there is anything else you want to talk about. It's the best way to create an issue here on GitHub for a proposal. If people mention things like this in a sarcastic tone on Discord then those messages will most likely get lost. I have actually planned to make a new slash command that should encourage more useful feedback.

Summary:

• Patch release every 2 weeks (or immediately if a regression happens)
• Minor release every 4 weeks (or immediately if a game update breaks everything)
• Major release when all planned features/bugfixes have been merged
• No more pre-releases

Update command changes:

• Add option to select version
• Add "canary" as version option

---

@ThisAMJ You are an admin for this repository so you have the power. If something is not stable you should put it behind a command. Changing the default behaviour of a command should also be avoided in the future. For patch releases you don't actually have to wait for a 2nd maintainer.

Here is the release workflow that I follow:

• Checkout master branch
• Double check the latest commits for potential bugs
• Check if there are no local changes that can affect the build
• Create a clean build
• Test if SAR loads and that it is the latest build
• Test if the game loads a map
• Test if the game disconnects to the main menu
• Test if the game exits
• git tag <version>
• git push origin <version>
• Wait for CD action to finish
• Replace TODO in release notes

[JaioCG]

Alright I guess I should probably type a little reply here since I pretty much went MIA after posting the original issue.

Firstly, I didn't intend for the main body of the issue to be targeted towards a single person, if anyone felt that way, or if someone felt it was targeted, I didn't mean for it to be, I deeply apologize for that.

Secondly, I blame a bit of my naiveness and lack of general research for me mentioning Semver. I knew the term from general "snooping around GitHub", and thought it was a general standard. I didn't want to mention Minecraft when it comes to versioning inspiration since I don't really like how it handles things like release candidates. But in terms of release formatting, yeah it would make sense to do version numbers similar to Minecraft JE or Beat Saber (maybe dropping the 1 though, not sure why many games still use it, i guess tradition?). I don't think I mentioned this is my main issue but I think the start of this "current downfall", if that's even the appropriate word to use, was 1.12.2.2. It's a funny update to laugh at for its versioning, but taking account what each digit means, it can sort of be rewritten as "SAR Community Version 2.2", since both the 1 and 12 are effectively useless when it comes to actual development. I'm not saying to rename SAR updates to "SAR Community Version", cause that's stupid, but I think the fear of long version numbers got us stuck to the weird release cycle we have today, when effectively half of it was not necessary information.

Thirdly, As for AMJ's reply here, I completely understand being unconfident for pushing updates, hell I do the same thing in mod-chat, which is part of the reason why Plays Games took so damn long to publish. But I think quick patch updates, (fixing crashes, etc.) shouldn't really have a thought behind it, and should be released as soon as possible. I get if that's still a little much, but without an update being full of features and having one set purpose, it should be a lot easier to 1) get a deadline set, and 2) actually releasing.

Fourthly (and finally), NeKz's reply has an idea for a release schedule, something I mentioned a little bit in my original issue but definitely wasn't the focus. As for releasing updates on a scheduled basis, I'm not quite how I feel about it. Mainly with sometimes a feature is made that players are very excited to play with or test out, having to wait X amount of time for the scheduled release to roll around may be a bit punishing. I think the current pre-releases are good enough just to honestly be renamed as minor updates, and of course patches when the updates are entirely fixes. The one good thing I can see about an active schedule though would be removing much of the sometimes months-long wait between some current pre-releases. As for switching versions and a canary feature, what would those be like? In my mind, it would be effectively the current CI builds, but in a public release format. I'm not really sure how I feel about that, mainly in a legality sense on leaderboards, but my biggest issue is that having two separate versions of SAR is something that I want to avoid with these changes, having a "Main" release and then a "Canary" separate release I think will eventually lead us back to where we are now, where "Stable" releases are very inactive and "Pre" releases are the ones being consistently updated.

[JaioCG]

Ok I will admit, one really nice thing about pre-releases is that right now, new features and fixes aren't really tested lol. Two recent examples come as the sar_disable_challenge_stats_hud and point_viewcontrol changes, both having very drastic side effects that weren't really tested before release. In this state, having these quick releases is very nice, but I feel in the future more testing should be put into place before releasing an any update of some kind.

edit: typos

[Alex-Marek]

I honestly think we should remove pre-releases entirely and the philosophy that comes with it. Every new commit that lands in the master branch and that is not part of a release is basically a pre-release. If people want to try out the latest changes they should be able to use the update command without waiting for any release. I'm not exactly sure how this would work but for future references I will call this a "canary release".

I think all the changes from the summary of Nekz's post are good, but I am a bit concerned about the killing pre-releases part. The main benefits I see of pre-releases are:

• They let us have a semi-legal ground where changes can be tested prior to being fully verified as legal.
• They serve as a bit of a buffer for making sure aspects of a feature are better understood/implemented before people even have a version to test them on. I'm almost entirely sure we've had a situation where a pre-release was about to be sent without a command cheat protected that should have been and just a bit before the pre-release happened someone said that the command should probably be cheat protected so people don't accidentally abuse it.
• Makes it less likely we have to do commit/version specific bans. These would be necessary in the instance where something gets fixed in a later Canary version but people just downgrade to the broken (presumably cheaty) version.
• They compile many different updates into one making it easier for verifiers/moderators/admins to identify what version a run was done on which could be useful if a bug is found to have caused problems for all the versions between x date and y date.
• Due to the less amount of versions as talked about in the previous point they work better with the mdp verification tool. With a legal canary version verifiers would have to update the sar_whitelist in the tool significantly more often.

Having the legal canary version does substitute nicely in for the first bullet point, but I don't think it works as well for the others (especially the third point).
The canary version does seem like it could still be useful as a non-legal version. I think part of the reason we had so many problems getting people to test non-legal SAR versions was because at the time you'd either have to compile it yourself or have it sent to you for a manual install. A non-legal canary version put together with your idea of having the update command take version parameters resolves that issue since switching back and forth is only one console command and a game relaunch away.

[JaioCG]

I think a non-legal canary version is the best way to go, as currently testing isn't really something that happens with the current pre-releases. With the two latest issues features were shipped out without much testing, causing major issues to be shown. Having a non-legal canary version with an easy update command makes it easy for people to test new features or bug fixes before being shipped out for a legal release.