[Netplay] Dealing with core backwards incompatibility #14149
Comments
|
@Chaos81 You've a whole community that netplay solely on RetroArch. You may want to express your opinion here. |
|
Well I would suggest a new fork for netplay that the version number can be bumped. Seems the dev of this core has very little interest in the projects netplays compatibility for the userbase and that is fine, The project as a whole needs to address this with a new unbiased fork that bumps up stable when needed if a compromise can not be met in situations like this. Either that or remove the cores netplay compatible status as it no longer fits this description for the project needs as a whole. I wouldnt throw all your improvements away for one core. Let the maintainer of this core make the choice and dont stress yourself. Someone else will no doubt make a netplay fork if the original maintainer has no interest. |
It's not just one core; the way it's now, any core can break netplay backwards compatibility without any warning. It just happened to be fbneo at this time. As for the fork, afaik, he doesn't want that either. |
Addressing this before it leads to drama. The reason I would stop (more like go into another hiatus) is because I can't keep netplay on a working state like this. Any improvements will be shadowed by netplay simply not working and if you think logically, this is a waste of a developer's time. |
Im pretty sure other cores would be willing to compromise. You are right though this wont work with some kind of versioning infrastructure. I seen this as an issue on the other thready of badly compromising backwards compatibility. I think youve have tried all your avenues in dealing with this core dev might be better talking to the libretro team rather than core dev for a big picture solution. Regarding his views on a fork its open source/source available whats best for the project will happen regardless I would imagine. The people at the top can make a policy on the future of netplay. If this keeps up people will just use things like fightcade that do make netplay a reality. |
Wouldn't be any different from a stable repository for cores. They would simply stop supporting libretro aswell as the possibility of bad relations between libretro and another emulator team, which I don't want. |
I can completely understand your frustration. The simple solution you suggested is workable stable releases for netplay and nightly dont work on netplay. This will stabilise netplay for the userbase as a whole. I think you both have different goals and only one wants to compromise. I sure the other dev will see the big picture he probably feels its something he wanted but didnt think the side effects to user base as whole and feels he opinions are not validated. When its really a big picture thing. Asking devs to bump the version monthly or so is not a big ask. |
Well this is true incompatible cores with no versioning just wont work. If devs cant update something as simple as a version on changes that effect netplay it might be worth removing these cores from netplay capable status and just keep the ones that make netplay a priority work in the lobbys. That way at least is doesn't give the impression of netplay is broken and devs can do what they please with the core as far as supporting netplay goes. |
|
Some platforms don't really have the luxury of doing that. Tagging cores as not netplay reliable but not outright locking them out of netplay is better than simply removing those cores from netplay, although as said before, platforms which can't use the core downloader/updater will be in trouble, just as it's right now. |
|
The biggest problem I see with any other solution than a stable version to work from is people will have problems. Example if you use commit hashes you cant join servers with different hashes. If the person hosting an older hash than you how do you get that version. Without a baseline to work from it just wont work long term and users will get the impression its broke like they do now. Since we cant get a baseline with this core per say the version rarely changes its not a great candidate for netplay support. A certain critera should be set if you want to support net play claims. I cant see netplay going forward without some intervention on implementation conditions to support it. I also dont think this is something you can fix it has to be a platform/team choice its not fair for this to fall on your lap with every dev. This issue goes beyond netplay though seems savestates for this core will be unstable for users. Unless it checks the size and deletes any incompatible ones to stop any issues occurring. |
Aye, which is why I've created this issue as I can't solve it myself. It falls on either libretro's infrastructure (core downloader/updater) or the cores themselves, both of which I've no saying in the matter. |
|
Well hopefully something fruitful comes of this in the long run. Either way you will have your answer in time . I think for this core in question and the devs feelings on the matter it has no future with netplay in a workable way for the user base. Hopefully other core maintainers see things differently and the platform can let people get a better better experience from netplay. |
|
I already explained it all in the other issue :
As a matter of fact, we can't provide support if a broken version of our emulator is frozen in time on libretro's side. This is the direct consequence of the policy change you are requesting. Hanging around libretro github/reddit/discord/forum just to tell users "thanks for the report, this is a bug on our side, we will fix it asap but you won't be able to download the fixed core anytime soon without first messing around with your
Your proposition also affects all non-netplay users so maybe you should start caring about them. Simply said, this "frozen core repo" solution you are proposing is causing major issues without actually guaranteeing the original problem of mismatching core versions is solved. |
I actually couldn't care less at the condition this fork doesn't become a burden for our team, meaning it must not be named in a way that would bring confusion and redirect all support to our team while it's actually not maintained by us. Ofc, i'd need to confirm with the other members of the team if they are ok with this. |
|
How exactly is having two different repos "affecting" other users? If you don't want to use the stable repo, use the nightly, It's literally a setting within RetroArch called "Buildbot Cores URL". Quoting this again for emphasis:
On a whim means a sudden decision, which applies to breaking savestate support for older versions without incrementing the version number. And this passive-aggressive condescending attitude is getting old. I just saw how you treated @MistyDreams at the other issue (and this is the second time I've seen you treating him like that). |
|
I already explained in length, in this issue and the other, why this "frozen core repo" actually doesn't really solve anything, and why incrementing version in the way you want is unrealistic and wouldn't be helpful.
This "frozen core repo" (stop calling it stable, it's not) is a trick only helpful for your netplay mismatch issues, and shouldn't become the default repository for downloading cores.
We have worked for years on this project, willingly improving netplay support along the line, building a user base who trusts us in improving our emulator on a daily basis. And now some people come and say "you don't care enough about netplay, everything should be centered around netplay, by default we'll freeze versions for everyone because it might serve the needs of our netplay community, and if your users aren't happy with that you can always tell them to change the default configuration". Sure, i'm totally the person in the wrong here. I'll stop there, it doesn't seem you can understand what i'm explaining. |
|
No need to get heated. We're all acting in good faith, and we all want everything to work as effectively and seamlessly as possible. We already have a solution to this issue, which is: stable releases with core snapshots for those releases: http://buildbot.libretro.com/stable/1.10.3/windows/x86_64/RetroArch_cores.7z Those people for whom netplay is their top priority can use these stable snapshots, as they will never change, and they should match the stable releases for other platforms for cross-platform play. |
And I don't mean to be rude, but if the fix was that simple, I wouldn't have opened this issue. TL;DR: netplay with currently wip cores is not reliable at all and this isn't a problem that can be fixed through code. Forcing core downloads won't work for platforms which don't allow it, like Steam. Guaranteening that cores on the same stable version be compatible on multiple devices, should be standard imo, but some people don't seem to understand the need for standardization and the consequences for lacking it. |
|
It's also ironic that I got a lot of shit from some contributors for wanting to break backwards compatibility between 1.9.X and 1.10.X to avoid shitty hacks and some inherently problems with supporting old broken code; but apparently, this (which can break backwards compatibility every few days) gets a pass... infuriating. |
|
Let's please find a way to compromise here so that neither side here gets alienated and neither side leaves development. We cannot afford any developers to leave over this. We have to find a compromise if only for the sake of the community and the users. Let's please find common ground here. |
|
@cthulhu-throwaway We cannot force a particular way of doing things suddenly on the FB Neo team. We have to find a compromise. |
|
I'll repeat again - throwing down ultimatums that one leaves development over a design decision is not productive on either side, and it is unfair and impossible for any of us to start showing favoritism to either side. If something is really that disruptive, the idea should be dropped, because coding decisions in no way can result in crucial developers having to leave or butt heads. We have to compromise then to let peace prevail. There is no point in chasing perfection if it's going to kill off a good amount of contributors in our current ecosystem. This is a team-based effort and decisions have to be made that the majority can agree on and can work with. Alienating people is not the solution. For what it's worth, I don't really like the idea of some 'netplay core repository' or whatever is being suggested here. I'd suggest we stay with what we currently have and we don't try to overegg the pudding here unnecessarily. It really is not that serious to warrant butting heads with one another. you have done excellent work and nobody is denying this. Let's take a step back though and not create a huge issue here unnecessarily when there doesn't need to be. We are seriously exaggerating any actual issue here. |
I agree with this, this is more than enough. We really don't need to resort to some controversial solution for nightlies and then risk developers from either side getting upset over it. If it's that controversial then we should not pursue it. Furthermore, the team is already overemburdened with enough stuff to do, and having to add even more to our workload to setup new workflows is not really something that is currently in our interest. If anything, we want less technical debt. |
Honestly a far better solution is just having something so that before a netplay session is started, if internet connectivity is available, it checks the version, and if it's out of date, it attempts to just download the newest core. It should be easy to just fire off a task for that. That is far more productive than creating some 'stable netplay core version' repository or whatever is suggested here. This is way too overcomplicated and we really cannot be asked to maintain all of that, it is yet even more laborous work and we already have way too much technical debt as is. From what I hear, @barbudreadmon would be down for this solution and it would solve the entire impasse right now. Let's not needlessly overcomplicate things here. The best-case solution if a version is out of date is to simply update the core, period. There is not enough manpower or labor force in this project to have to maintain lists of 'stable' versions for netplay, people are assumed to either be on stable core versions or on the latest nightly cores. Anything else is going to be a total maintenance disaster/nightmare and something that there simply is no manpower or labor power for. |
I'd suggest instead of creating a scary warning message, have the tag in the info file, HOWEVER, instead of doing a scary warning, have it automatically update the core on the spot then for platforms where this is possible (should be possible for both statically linked and dynamically linked targets I think when online updater facilities are available). It should be as easy as triggering a HTTP download task and then extracting the core to the appropriate directory, there's already functionality for this existing. You'll have to come up with something else for Steam though which will require you to talk to @Mats since he runs the Mist integration part of the Steam version. Anyway, I'm afraid all of this is going to result in tons more work either way. I'd suggest we hold our horses right now, it really is not much to ask users to just make sure they are on the same core version. That should be expected and there is a handy and easy 'update all cores' button exactly for this purpose. |
|
Dude, it's not ambition, it's literally in the first post that I can't join 1.10.3 (latest stable) rooms running fbneo because I am running a newer commit of fbneo that isn't backwards compatible with something as short as 2 months, but is the only one available through the core downloader. And it's not an ultimatum; until now I assumed those cores remained backwards compatible on the same version number, now I know better and I can't continue working on netplay until this is solved. Also, it's not me that needs to compromise, I can't do anything; this falls beyond the netplay code in RetroArch as stated multiple times. Already went through the core download issues on platforms that won't allow it. So, not much else for me to say other than to read my previous comments carefully. And again, and you know this very well, as you were one of the people giving me a lot of shit for a single break in backwards compatibility, how come this gets a pass just like that? Sure, you can't tell them what to do with their own core, but you can make better decisions than them with your infrastructure. Finally, I am not just the current main contributor to netplay, but also an active netplayer; people will often find me joining their rooms. I know most of the pros and cons of the system by now and if I am saying I can't continue working like this, it means it's a huge issue that I can't fix on my end. |
Because FB Neo developers would leave over this (we were told). And conflict and drama over what? Let's just compromise. It is really not all that serious. It simply isn't possible for them to maintain that kind of compatibility over months. It is a project worked on by many people, they don't all have the same interests and only a handful of devs are involved in the libretro core side of things like barbudreadmon. The only workable approach there is just updating the core on both ends to the very latest version. That is the only workable solution for them there, and it's understandable. Would also encourage you to please come on Discord so we can talk there. Things tend to get too heated on Github and it is really not productive to getting these impasses solved. Already suggested earlier meeting up there since there is still the relay issue. You can contact 'BoardsOfCanada' or 'hunterk' there. There doesn't need to be all this drama over something that ultimately we can find a compromised solution for that doesn't involve having any developers leave, and we can just be on our merry way continuing development as usual. We prefer if we could resolve this on Discord in DMs. This is not the proper venue to solve this. |
|
There is another bad decision, which is centralizing all of your communications on a platform like Discord. I am not going back there, man. |
|
Also, I didn't complain about them having to maintain backwards compatibility, but rather their lack of version increment on such occasions and such commits getting pushed to the core downloader automatically, without an alternative that doesn't involve non-standard approachs (aka going to the buildbot url and download cores manually). How I am supposed to keep netplay functional in an environment like this? |
Luckily for me, I make the community download a specific RA package with certain features disabled, like the core updater/downloader. This way I know everyone is on the same core version. And I don't usually update the RA version unless there are some netplay or other improvements that benefit us, as it's a pain to get everyone to update. Unfortunately, you can't do something like that with something so open to everyone. So you are going to run into these issues. The warning messages that pop up about core version mismatch are usually ignored. What about forcing a download of the matching core on connect? For those systems that can't download a core on connect, maybe adding a way to search the lobby for games with matching cores (or compatible core versions, which would require a list of versions compatible and might be too much work for developers)? Also, like to note Cthulhu has done an amazing job on netplay. It's been working better than ever. |
I never considered you a moron, you were very helpful just the other day with the savestate context PR, i certainly freaked out about the "frozen core version" proposition though, since it would totally break how my team interacts with users. You made many valid statements in this issue that indeed need to be addressed. |
|
Just took a look and this is a whole lot of work (as expected).
If this was simpler I wouldn't mind giving this a try, but so much trouble for something that I am not fully in agreement... I think there is also a need for an internet connectivity check in order to skip core updates instead of erroring it out on the user. |
|
The only potential issue I can see is if a core is updated just before a user joins and a session is already going on with an older version there would potentially be compatibility issues on joining. I think a commit hash check would avoid this though. That about the only input I have to add. |
|
Production-grade netplay platforms would make the service unavailable for the duration of the update/maintainance, this is neither something we could do (players are directly connecting to each other), neither something we should do (an update is not necessarily breaking netplay backward compatibility on the game in use). A notification that the versions are mismatched would be sufficient, possibly mentioning that the host is the one running the older version to avoid confusion for the client running the newer version. |
|
if its the case versions are mismatched would be sufficient, everything is ok there is no real issue with the way things are now surely. What do you feel we would gain from the changes you all suggested so I can actually understand the gains here. I think this is the real difference of opinion for me anyway it does matter if you hit the 5% more than you would like because people lock there old cores, newer core connect with failure. This is what allowing mismatch and not dealing with locked cores as an non issue leads too (two ops iterated this was a non issue). Again if its a live with situation for you guys, It really is what it is. There would be no point in even debating it at this point. I really dont get how a user base must have access to latest cores but netplay can use any old one. The only seriously obtainable target is master for some kind of consistency even that wouldnt be obtainable as mismatch is encouraged and its a 50/50 if mismatched cores are compatible. This is why I agree its not a healthy state as it could be as is imho. |
Such notification already exists and is hidden/ignored by most (including me). Verifying which version is the oldest is not possible; 1.0.0.03 [older commit hash] can't really be compared arithmetically with 1.0.0.03 [newer commit hash]. |
It might be because it's too common of an occurence at the moment.
Indeed, to order 2 different commits, we'd need to either update every core to use the |
|
A way to simplify core updates is to hook the core initialization routine and just perform a core update (if netplay is enabled) before loading the core. EDIT: This is likely the place to do it: Lines 1727 to 1751 in 882829c and you can use a task there. Still requires to write a single core update task and special code for a netplay host already running content. |
I thought about it again, this is actually much simpler than that, and doesn't really require something like a database or version incrementation at each commit :
|
|
should the host do a check (when possible) against the latest and get a warning, too? "you're not using the latest core version. Clients may have trouble joining"? |
|
Found this branching for the core updater. I know plenty of people netplaying through their xbox, there is no core updating there aswell? |
|
There's not really an obvious core repository to point to. Maybe for Xbox you could get away with just pointing to the regular x64 Windows cores, but for ARM Windows 10 phones that obviously wouldn't work, and we don't really build cores for that either. Anyway, that needs more thought. Best not to worry about this particular platform for now, don't let it hold you back from anything discussed in this thread. |
ghost
mentioned this issue
|
At least half of the brazilians netplay on xbox... This forced core update will wreak havoc on them. |
|
The real problematic side of this is if a group of people decide to lock old cores and host intentionally and youll have issues. Most people are good but if you get a hostile intention your netplay can be disrupted very easily. This is not pertaining to this update it has always been the case to allow mismatches. |
This certainly would be helpful for people if an update occurs in a session already running. Locked cores should get denied hosting to avoid potential innocent/intentional disruption to other players, or at the very least taken under consideration to do so. There is no real version checks or any kind of baseline for compatibility matching this would mitigate any problems escalating though and keep things aligned to current master as some sort of baseline. Cores that update multiple times a day will be sending this message regularly but thats the nature of moving targets as your baseline. |
|
Not possible to block locked cores without a version database accessible by the lobby server. Even then, that's not a good idea because there are communities running specific builds of RetroArchs without the updater. |
|
Im really at a loss how to fix this with the current model selected to work with and the lack of resolve to reach a baseline of some sort. It would indeed be wrong to punish communities that do create a working baseline without an updater. I guess plug and pray it is for now at least. |
It should be disabled for now then on Xbox until we expose the core updater there. |
As far as I know, it's already disabled for it, but that's not the problem. |
|
I don't know much about the xbox situation, it'd be ideal if they had an online updater too indeed. |
ghost
mentioned this issue
|
The following platforms seem to be a problem for the auto update to latest version feature: Wii, 3DS, Vita, tvOS. Some project files for iOS don't define HAVE_UPDATE_CORES either, which might be a problem if those are currently active. |
|
FWIW, wii & 3ds aren't gonna be a major issue as far as FBNeo is concerned, FBNeo is unavailable/unusable there due to the memory limitations of those devices. I made a special 3ds build with a reduced list of supported games but that didn't really resolve the whole issue (the neogeo system is included but only a handful of neogeo games can fit into memory), it also seems retroarch is somehow leaking memory on 3ds since they can't load games consecutively, so i'm still considering removing the core there entirely. Forget about wii, the limitation there is even worse and requires split cores heavily modified for wii usage, the fbalpha2012 neogeo/cps1/cps2 cores are handling that job. I don't know much about FBNeo usage on the other 2 platforms. I'm kinda surprised netplay is available on those 3 console devices, i'd expect the overhead from netplay to make things even more unplayable there due to the already low performance in a normal context. Also note that the wii endianness is probably a far bigger issue than versioning as far as cross-platform netplay is concerned. |
|
Netplay code has specific guards for endianess (or platform-specific cores): https://github.com/libretro/RetroArch/blob/master/network/netplay/netplay_frontend.c#L1033-L1059. Still, one could netplay with other matching platforms or by playing with non-quirk cores. RetroArch isn't just leaking memory for the 3DS build, it's leaking everywhere (the joys of C for large scale projects). I've lost count of how many memory leaks I've fixed within netplay-related code alone. |
|
While nobody is discounting that there are and might still be memory leaks in RetroArch (although if there are, we'd like to see evidence of it so we can do something about it), it's important to keep in mind that many standalone emulators (and by extension the libretro core ports) leak memory too, they usually just don't bother fixing the leaks because the leaks go away when unloading the executable anyway and/or because they don't have the same tight RAM/heap constraints as say consoles. These are just guesses of course but we've seen it happen often enough. In our hard forked emulator repos the likes of jdgleaver plugged many memory leaks that were present in standalone emulators. So always take into consideration it's not always just RetroArch, many times it's the core itself and often times it's a memory leak present in standalone too. Other times it could be an implementation issue to do with the libretro core. We of course very much want to see any and all memory leak fixed though. So whatever contributors can do to help solve them, it's all very welcome. |
Description
See: libretro/FBNeo#996
I've been unable to netplay fbneo for the past few days because the hosts are running RetroArch 1.10.3 stable but fbneo versions prior to the one that breaks backwards compatibility (nightly).
These players are randoms that I just find in the public lobby, simply telling them to update the core isn't possible.
RetroArch netplay's has multiple advantages over other rollback-based services, with its main strength being its support for more than 2 players.
However, with cores not guaranteeing compatibility within the same version (same version number, not same commit), RetroArch's netplay for those cores is not reliable at all.
Expected behavior
Cores should remain backwards compatible for some time, if the version isn't incremented.
Actual behavior
We only have unstable cores in the core downloader/updater, therefore, backwards compatibility is compromised on cores that get worked frequently.
Side notes
I've been working on networks since the early 2000s, and I've never worked on a project where network enabled components were distributed to the general public straight from unstable.
We often had alpha and/or betas (closed and open) before we would push that update to stable and thus, to the general population.
Problems (as shown here) and security leaks are two things you can expect to encounter if you don't follow with these.
@LibretroAdmin You argued a lot with me about keeping netplay backwards compatible, even when hacks were necessary to do so. But this is effectively pointless if cores can and will break it on a whim.
I've been working tirelessly to improve netplay, but I'll stop if this issue is not addressed.
Solutions include the already mentioned stable repository for cores, or a tag in the info files that allow me to display a very visible notification that the core is not reliable for netplay and should always be updated before hosting/joining (this will be a huge problem for platforms which can't update cores from the unstable repository).
It's also really important for netplay users to express their thoughts on this as it affects them all.
RetroArch's netplay seems to always be treated as a second-class citizen and this needs to stop, or deprecate it and recommend other services.
The text was updated successfully, but these errors were encountered: