Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suggestion - Please make a Linux native version #2862

Open
VortexFeedback opened this issue Jan 8, 2019 · 34 comments

Comments

Projects
None yet
@VortexFeedback
Copy link
Collaborator

commented Jan 8, 2019

Vortex Version: 0.16.15
Memory: 15.61 GB
System: win32 x64 (6.1.7601)
Please make a Linux native version so that I don't have to just run it through WINE.

Reported by: AmayaSasaki

@8BitCerberus

This comment has been minimized.

Copy link

commented Jan 12, 2019

Especially with Valve's Proton/Steam Play now getting so many games, including many serviced by Nexus' mods, running as or nearly as easy as on Windows. And for non-Steam games there's Lutris that takes a lot of the hassle out of managing Windows games on Linux.

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented Jan 14, 2019

Vortex is built on technologies that are also available on Linux (and MacOS), except for a few native libraries that would have to be re-implemented (stuff like reading the application version from a .exe file).
It's mostly a matter of investing the time, something we can't do at the moment.

@TheMatthew

This comment has been minimized.

Copy link

commented Jan 20, 2019

I was looking through the code "lightly" I don't pretend to know what I'm doing/saying.

It appears as though the only "windows" dependency left is Nexus/wholocks which could be replaced by https://www.npmjs.com/package/@ronomon/opened or something similar.

Everything else at the moment looks OK.

I am not sure when I will be competent enough to make a PR. if there are any Node/typescript masters out there, this could be a cool feature.

@VortexFeedback

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 7, 2019

Vortex Version: 0.16.15
Memory: 15.66 GB
System: win32 x64 (6.1.7601)
I see Vortex is an electron app, with the improvements being made to Proton/Steam Play it would be incredible if there were a native Linux version of Vortex to use with Steam Play games.

Or at the very least some documentation put into getting every thing up and running under Linux, since Vortex does seem to run fairly well under WINE.

Reported by: Dreae

@TheMatthew

This comment has been minimized.

Copy link

commented Feb 7, 2019

I doubt that steam play / wine would work well on vortex as it uses undocumented windows apis. Just a heads up.

@8BitCerberus

This comment has been minimized.

Copy link

commented Feb 8, 2019

It works very well, actually. You can either install it directly in your game's prefix to manage that game (and subsequently install it in to each game's individual prefix that you want to manage), or set up it's own prefix and use symlinks to point it at each game install. I've currently got it managing Skyrim SE and installing mods is no problem. The only real issue is SKSE64 (and F4SE) doesn't work under Proton or Wine yet. There is a patch you can use and manually compile both Wine and Proton to get the script extenders working. But Vortex itself works pretty much flawlessly.

Since Vortex is using sym/hardlinks instead of MOs virtual file system, a native Linux version should just need to know where the games are installed and use Linux's link command instead of Windows' (ln vs. mklink).

@TheMatthew

This comment has been minimized.

Copy link

commented Mar 3, 2019

There's a magnificent individual that did this Nexus-Mods/node-wholocks#2 I got vortex working in wine, so it "works" but a native version would still be an interesting prospect for some.

@Mumrik93

This comment has been minimized.

Copy link

commented Apr 9, 2019

Getting Vortex to work on Linux would really kick up Linux as a viable alternative for gamers who are tired of Windows.. of which there are many, myself included.

@TanninOne TanninOne added this to the tbd milestone Apr 24, 2019

@codeman101

This comment has been minimized.

Copy link

commented May 5, 2019

@TanninOne

I'd love to assist this issue and devote time to solving it but a hurtle I'd have to overcome that I'm not willing to do at the moment is going through the process of learning TypeScript. I'm curious as to why that was decided to be the base language of the project vs C/C++, C#, Java, Python ect?

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented May 6, 2019

TypeScript is a much easier language to learn than C++, C# or Java and less "unusual" syntax wise than python.
The decision was made
a) to encourage contributions from less experienced developers because if you already know one of the above mentioned languages you can start writing useful typescript code within hours
b) because a customisable UI was one of the top requirements for the project and HTML/CSS (vortex is based on electron) is more readily customisable and better known to more people than something like QML or XAML.
c) If this project had been written in C# (which was in fact an option we discussed), it would be using WPF and XAML and a Linux port would be entirely out of the question. If it had been written in C++ (another option) there would be a lot more code that would have to be ported for Linux (or MacOS). Java is never a good choice for UI-heavy, close-to-the-system applications (A lot of what Vortex does is managing files, creating symlinks, fixing permissions and such, I wouldn't want to do all that through JVM...). And python - what would be the advantage of python over javascript?

@codeman101

This comment has been minimized.

Copy link

commented May 6, 2019

A few things.

  1. As far as python over javascript I'd think the syntax is easier for a beginner. (but I agree aside from Qt as of recently, there's no simple framework to write a GUI with python)
  2. If the reason the project wasn't made in C# was because it'd be very hard to bring over to Linux why wasn't it designed with the intention of bringing over to Linux in the first place?
  3. If the project is based off electron why is it using typescript instead of javascript? I thought all electron based apps used javascript for their UI and by their nature were cross-platform (for example slack) I even watched a youtube video a while back making it sound like the big plus to making an electron based app was that it would automatically be cross-platform. So you can imagine my confusion.

Update As far my third point I started going through the list on electron wiki of electron based apps and realized their not all javascript based. I'll add it to my to-do-list to try to help with making this project Linux native. :)

@Patricol

This comment has been minimized.

Copy link

commented May 7, 2019

@TanninOne

I'd also be interested in working on this; but unless I can figure out which components are os-specific without needing to read through the whole project; it'll be hard to prioritize.

If you can point out any of the relevant areas in the code; it'd greatly lower the barrier to contributing.

Or potentially add inline markers (e.g. //TODO: fix for linux) noting os-specific things as you run into them during the work you're already doing on the project. Or any other thing you'd prefer to do that'd achieve a similar result.

@codeman101

This comment has been minimized.

Copy link

commented May 7, 2019

@TanninOne

In addition to what @Patricol said can you confirm whether or not having a Windows system is a requirement to figure out some stuff? (I don't run one anymore other than wine)

@Patricol

This comment has been minimized.

Copy link

commented May 7, 2019

@codeman101 We'd need to test that our changes don't break the Windows version, so I'd say yes.

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented May 7, 2019

  • Python may be easier if you know no programming languages at all, but as soon as you know any language with C inspired syntax, JS/TS will be easier
  • We don't currently have the resources to support Linux but we always made it a point to don't burn down any bridges in regards to Linux support.
  • typescript is more of a "meta" language that gets compiled to javascript at build time, all it does is provide us with type checking during build time. For an extension developer it doesn't matter whether you use javascript, typescript, coffescript, probably also stuff like elm or whatever. All languages that transpile to javascript are fine.
  • slack is cross-platform because it uses no/practically no platform dependent functionality. Vortex is by nature closer to the system and more performance critical. It is a much more complex project and much closer to the OS. For example: Slack has no need to access the windows registry (directly).

A few examples of how Vortex might require and use platform dependent code:

  • Vortex does require using the registry, because it's the quickest, most reliable way (on windows) to figure out where the games are installed.
    Electron is a cross-platform toolkit but it doesn't provide functionality to access the registry (obviously).
    Our only choices here were to either limit the usefulness of Vortex on our main platform or write platform dependent code.
    However, to my knowledge we put all code accessing the registry is behind a platform check and so Vortex wouldn't attempt to use it on Linux, instead using a fallback mechanism which would work on all platforms (either parsing Steam configuration files or searching the disk).
    But we don't know, because we didn't test if
    a) Vortex actually looks for the steam configuration files in the right place on Linux
    b) Whether the steam configuration files are actually the exact same format on Linux
    c) Whether Vortex correctly determines automatically the correct search path for games

  • electron (or rather node.js) provide a cross-platform way to read directories but the way it works when you want to read recursively is excruciatingly slow on windows because the cross-platform ui emulates posix api where you have to readdir, use stat for each entry to determine if its a directory, if so read that directory as well,...
    The native windows equivalent to readdir already provides the information which entry is a directory (nodejs just drops that info) so all those stat calls (which are probably more expensive on Windows) are unnecessary.
    Because we have to read directories so often with Vortex we use our own library (turbowalk) that wraps the faster windows api directly, and falls back to the readdir/stat code on other platforms.
    So we already have provisions to support Linux, they just aren't tested and they may not support the same options as the windows variant (stuff like "follow symlinks or not" or "list hidden files or not") which might then manifest in subtle (or not so subtle) bugs when running Vortex in Linux.

Tbh.: My advice to anyone trying to work on a Linux version: Start by setting up a fork for working on Linux compatibility, start by trying to build it. Investigate build errors if there are any, fix them in a way that hopefully doesn't affect the windows build, when you have a running build, test the stuff that is intuitively more low-level (stuff like getting at the icons for executables, searching disk, creating links)
Don't start looking at the code trying to figure out what needs to be done.
Once you have it working natively on Linux, see if you can reduce the difference to the official build and then we can check together if anything might affect Windows compatibility before merging it into the official fork.

Also: the parts that are most likely require work are native modules we created, not the Vortex repo itself.
look at the dependencies in package.json, our custom modules are the ones that aren't listed with a version number but with an identifier for a github repo.
Stuff like

  • Nexus-Mods/node-crash-dump
  • Nexus-Mods/node-exe-version
  • Nexus-Mods/node-icon-extract
  • Nexus-Mods/node-permissions
  • Nexus-Mods/node-turbowalk
  • Nexus-Mods/vortex-parse-ini
  • Nexus-Mods/node-wholocks

are bound to require platform-specific code that is either not implemented for Linux or not tested.

@codeman101

This comment has been minimized.

Copy link

commented May 7, 2019

@TanninOne

Tbh.: My advice to anyone trying to work on a Linux version: Start by setting up a fork for working on Linux compatibility, start by trying to build it. Investigate build errors if there are any, fix them in a way that hopefully doesn't affect the windows build, when you have a running build, test the stuff that is intuitively more low-level (stuff like getting at the icons for executables, searching disk, creating links)
Don't start looking at the code trying to figure out what needs to be done.
Once you have it working natively on Linux, see if you can reduce the difference to the official build and then we can check together if anything might affect Windows compatibility before merging it into the official fork.

I made a fork of the project last night. Although at the time it was out of anger over the comment Patricol made in response to my question about needing to run windows. Although now thinking about it more I suppose I can run a VM of a windows preview build to test it.

Due to the fact that the program is using the registry to detect the games the only way I can see to make it cross-platform is to have the program detect which OS it's running and do a recursive search of the users home directory for the Linux implementation.

Edit: Also you claim that having the program use the registry to find games is faster but I just remembered from when I ran windows that although that may be true it didn't get all of the games.

I realize now that the design trade-off is "should we do an implementation that would make game searching really fast and use something that is windows dependent or do an implementation that would be slower but more cross-platform friendly." Personally I don't feel the the correct tade-off decision was made for two reasons.

  1. The current implementation of the registry method doesn't get all of the games
  2. I never considered the game search method to be much of a plus over vortex's predecessor. The main plus I realized with vortex over it's predecessor is the ease in the way mods are added. The slowness in the search didn't bother me nearly as much compared to how mods were added.
@TanninOne

This comment has been minimized.

Copy link
Contributor

commented May 8, 2019

@codeman101 we're not debating design decisions here. Just let me make one thing very clear here: I'm open to a Linux version but no trade-offs will be accepted for the Windows version. 0. Zilch.

second: Please educate yourself. Seriously, it's getting pretty annoying discussing with you because you insist on discussing about things you don't fully understand.

Vortex has custom detection code for all games, usually with fallbacks. Every game that Vortex tries to detect through the registry does get detected immediately (within less than a millisecond), the only people having problems with that are those that insist on mucking about manually in their registry or move games around manually instead of doing it properly.
So 99% of our users get all their games discovered within milliseconds, a full disk search like NMM did (which is also an option in Vortex!) can take hours on a large disk.
The claim that this is worse than just having a disk search is either misinformed or insane.
The fact you don't even seem to be aware that Vortex already supports searching the disk for games also shows you have spent very little time with the tool before coming here to tell us what to do.

You seem intend to construct an excuse for changing core functionality because you think it would make cross-platform support easier. a) It wouldn't b) it will not happen anyway.
Again: Windows is the primary platform, absolutely no concessions will be made to the Windows support just to make Linux support easier.
Either accept that and try to be constructive or please go away.

@Patricol

This comment has been minimized.

Copy link

commented May 8, 2019

Although at the time it was out of anger over the comment Patricol made in response to my question...

@codeman101 My apologies; I didn't intend for my last comment to be abrasive in any way.

@codeman101

This comment has been minimized.

Copy link

commented May 10, 2019

@Patricol

All good. I feel like some good has come out of me making my own fork because it's forcing me to learn the Windows API which has always been on my to do list anyway for the desire to help wine development.

@TanninOne

I agree. I will not argue with you anymore. Let's just agree that we have fundamental differences as to how we would've approached this project and move on. If I'm able to convert the necessary submodules to Linux code (which by the way aren't the ones you listed above) I'll post links to my changed files here. Unfortunately that's the best I'll be able to do as I copied a branch that makes the project ad-free. (one of the fundamental differences) Again I don't want to argue anymore I'm just explaining why my changes won't be mergable. Since this difference prevents my changes from being mergable anyway I'm going to try to covert the code into a pure Linux version (overwriting the Windows API code)

All and all I'm glad I forked this because as I said before it forces me to learn the windows API so even if someone beats me to punch of making this Linux compatible at least I got quite a learning experience out of it.

Oh also I'm sorry for what I said in the pull request. I didn't read the whole post of your first response and I've come to agree with you about that request.

@Patricol

This comment has been minimized.

Copy link

commented May 10, 2019

@codeman101

Just try to be more understanding. We're all human, and attacking people (and/or their views) left and right will get you nowhere in life; regardless of whether or not you are correct.

@codeman101

This comment has been minimized.

Copy link

commented May 11, 2019

@TanninOne

While trying to work on making the project Linux native I realized something and made this issue on proton (explained in issue) ValveSoftware/Proton#2670 and because of that I've decided to delete the fork I made since as far as I can tell it would only benefit native Linux games.

To future readers who still want to make this native for Linux

  • you need to install mono to get yarn install not to fail
  • Most if not all of work that needs to be converted from the Windows API are a set of a submodules in the Nexus github page. Aside from the one's TanninOne mentioned a few post above the others I've identified are
  • node-esptk
  • node-bsatk
  • node-ba2tk
    Each of them (including the one's mentioned a few post above I think) has a file called string-cast.h which seems to be the only file that needs to be redone and fortunately the code seems to be same (for the most part at least)

Search

node-

in the Nexus-mod github page to find all the modules that have this header file.

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented May 13, 2019

esptk, bsatk and ba2tk are all for gamebryo games (TES, Fallout) which don't run natively on Linux anyway.

Tbh if you're playing a game through wine on Linux, I would also run Vortex through wine and use native Vortex only for native games - things are just going to be too complicated otherwise.
For example: Vortex uses LOOT to order plugins for the gamebryo games. LOOT being a tool exclusive to windows games itself works under the assumption it's on windows - having stuff like case-insensitive file name comparisons hard-coded into it - same with the "gamebryo-plugin-management" vortex extension that uses it.
Another example: Vortex has to manipulate ini files for certain features (e.g. to have separate save directories for different profiles in gamebryo games). Now ini is not a standardized format, implementations differ in how they handle comments, multi-line strings, encodings and so on. Vortex uses a library called "vortex-parse-ini" with a plug-in interface to support different ini-styles. The only one implemented right now is the one used by the windows api and we use winapi functions directly to ensure perfect compatibility with games that also use those apis - like the gamebryo games.
On Linux you won't have those apis available, so you have to use some other ini format. Easy enough to implement but it may not be 100% compatible with the game, which could then lead to obscure bugs where ini settings don't stick or stuff like that.

I imagine there is a lot of grief down that rabbit hole.

@codeman101

This comment has been minimized.

Copy link

commented May 13, 2019

@TanninOne

esptk, bsatk and ba2tk are all for gamebryo games (TES, Fallout) which don't run natively on Linux anyway.

Oh ok.Well if someone tries to make a native Linux version of vortex they'll need to do something about them because yarn build fails because of them being there.

Tbh if you're playing a game through wine on Linux, I would also run Vortex through wine and use native Vortex only for native games - things are just going to be too complicated otherwise.

Yes I know but I didn't realize it until my previous post because I was actually trying to run TES via proton from the native steam client and then realized it'd make more sense to install vortex into that proton install of TES (hence the issue I linked to in previous post)

Another example: Vortex has to manipulate ini files for certain features (e.g. to have separate save directories for different profiles in gamebryo games). Now ini is not a standardized format, implementations differ in how they handle comments, multi-line strings, encodings and so on. Vortex uses a library called "vortex-parse-ini" with a plug-in interface to support different ini-styles. The only one implemented right now is the one used by the windows api and we use winapi functions directly to ensure perfect compatibility with games that also use those apis - like the gamebryo games.
On Linux you won't have those apis available, so you have to use some other ini format. Easy enough to implement but it may not be 100% compatible with the game, which could then lead to obscure bugs where ini settings don't stick or stuff like that.

Fortunately since the games aren't Linux native anyway this isn't an issue since you'd have to use a wine backbone to run the game so you can install the windows version of vortex into that bottle. (which is a feature being worked on for proton see link above)

Side note: I'm going to see if the wine database has reported this bug but I was reminded by my computer tonight that currently vortex doesn't run on wine past installation regardless of install the dotnet version mentioned in the one review on the database.

Edit: I made the bug report for it on the wine database.

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented May 13, 2019

esptk, bsatk, ba2tk and gamebryo-savegame are all libraries pulled in from extensions: gamebryo-plugin-management, gamebryo-savegame-management, gamebryo-archive-invalidation, gamebryo-...
I would disable those extensions entirely, that way it will not try to build the extensions either.

The best place to do that is probably in "BuildSubprojects.json" and "BuildSubprojects.js".
The js is the code, json is the data file which specifies which subprojects (mostly extensions) to build.
I would add a new "platform" parameter there, something like
platform: ['win32', 'linux']
Of course this would have to be implemented in the js file: "if build platform not in list of platforms, skip subproject"

Then, to start off, just mark all subprojects as windows only, try to build.
Get the core application to build and run on Linux, then check which extensions even make sense for native Linux and enable/fix them one-by-one.

@Freso

This comment has been minimized.

Copy link

commented Jun 21, 2019

LOOT being a tool exclusive to windows games itself works under the assumption it's on windows

As one of "the LOOT people", I’d just like to point out that LOOT actually has built Linux native binaries for years (available on Bintray). While it is true that it by default assumes Windows‐ness, it shouldn’t err out if not run on Linux and should be able to sort load orders for Wine/Proton installed games given the right settings. I’m not sure if we’ve actually had anyone do this yet, but it should theoretically be possible AFAIK. :)

(This also means it should be possible to also use LOOT(/loot-api) on a Linux‐native Vortex build, either for modding Proton-CK games or for something like OpenMW. If there are any issues with that on the LOOT side of things, please let us know!)

@ScumbagDog

This comment has been minimized.

Copy link

commented Jun 22, 2019

The best place to do that is probably in "BuildSubprojects.json" and "BuildSubprojects.js".
The js is the code, json is the data file which specifies which subprojects (mostly extensions) to build.
I would add a new "platform" parameter there, something like
platform: ['win32', 'linux']
Of course this would have to be implemented in the js file: "if build platform not in list of platforms, skip subproject"

Then, to start off, just mark all subprojects as windows only, try to build.
Get the core application to build and run on Linux, then check which extensions even make sense for native Linux and enable/fix them one-by-one.

I just tried doing exactly this, I actually got the main UI working. That being said, the dashboard threw this error when it tried to render, and I can't actually set a game mode (as well as a slew of minor issues).

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented Jun 23, 2019

Oh right, it should be safe to put the "setUserTasks" into a if (process.platform === 'win32') check, it's nothing important.

@Freso thanks for the info, I was actually not aware of the LOOT linux builds, sorry for the misinformation in my earlier post then.

@ScumbagDog

This comment has been minimized.

Copy link

commented Jun 23, 2019

I've stumbled upon another "interesting" issue. Running yarn run build doesn't make a proper binary in the out folder, and through debugging the dist command (which does produce a binary, but still throws errors in doing so, full error dump here.) I received the following error:

error TS5042: Option 'project' cannot be mixed with source files on a command line.

I'm not particularly well versed in JS (or TS) development, so I am not quite sure how to fix it, but I'm fairly certain there's an issue with the configuration of the app-builder-bin module.

Edit: It actually appears to just be tsc that doesn't like me putting the DEBUG= flag after it, so disregard that error message

@ScumbagDog

This comment has been minimized.

Copy link

commented Jun 25, 2019

This is a pretty decent start, though it isn't particularly usable yet. Do you happen to know what .net is used for, and if it even makes sense to try and use mono as a replacement?

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented Jun 25, 2019

.net is used for most mod installations, that is: determining which files to extract from the mod archives and where to put them, especially when installing mods with scripts (fomods).

There are two kinds of fomods: those based on xml files and those based on csharp scripts.

The xml based ones should work with mono. Not sure if the code needs any patches to Nexus-Mods/fomod-installer but it should be very doable. Of course one could also port that code to JS or something but that'd be a lot of work.

The csharp scripts however run essentially arbitrary code to create a ui, copy files and so on. These may be harder to get working with mono but more importantly they may be a security risk.
On windows (Vortex, NMM, MO) the script is run within a sandboxed AppDomain which limits file permissions the script has and so on to limit the damage a malicious or buggy script can do.
To my knowledge this sandboxing is not supported in mono and .net core (MS has stated that they consider it a mistake and insecure so it's unlikely they will add support to core in the future)

So my suggestion would be: Simply refuse to support c# scripted installers on linux. That's what I'd do. Or at the very least tell the user about the potentially security problem. Those installers are mostly used with fallout 3/nv and have come out of fashion with later games fortunately.

@ScumbagDog

This comment has been minimized.

Copy link

commented Jun 25, 2019

So my suggestion would be: Simply refuse to support c# scripted installers on linux.

I think this is a good start. Or rather, a good milestone after fixing the BSA-extensions (not completely sure how many of them work, but they use a windows-header in their code, which may or may not be horrible to patch out). As for the C# installers, mono does appear to have a sandbox, but I don't know how well it actually works, so focusing on XML-installers is definitely more lucrative to me.

@ScumbagDog

This comment has been minimized.

Copy link

commented Jun 25, 2019

On a sidenote: I get some errors when running the test suite (primarily in src/util//Steam.ts where app is undefined). Should I just ignore these for now and work on the broken extensions, or are they actually important?

@TanninOne

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2019

That looks like it could be what we need re mono sandbox, yes. Maybe I had outdated information about mono. But yes, xml installers are far more common. The only big mod I can think of for Skyrim that has a c# installer is skyui.

Regarding the unit tests: It depends on the issue but yes, I'd ignore it for now.
The problem with Steam.ts I think is that on windows it retrieves the steam config location from the registry, on other platforms it uses an electron function to determine the home directory and derives the path from that.
Unit tests aren't run in electron but in native node.js though, so that function isn't available.
However, we don't have unit tests to test the Steam functionality itself anyway, it's just imported as a "collateral".
Solution will probably be to create mocks for those functions because we don't want unit tests to access real system information anyway.

@elken

This comment has been minimized.

Copy link

commented Jul 4, 2019

Might be possible to use this library to execute C# scripts and wrap the scripts in mono sandbox.

As for unit tests I agree using a SteamProvider to mock would be the best solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.