Linux Support #1430

bwrsandman · 2019-08-14T10:54:02Z

Current State

Xenia on Linux compiles for Travis tests in order to check C++ errors, style and very basic functionality. It has some nice abstract function definition. Most of these definitions are stubs with failing asserts which mean running Xenia in debug causes SIGABRT and running in release will eventually crash due to missing implementation.

The biggest missing part is the threading. Without threading implementation, the Xenia UI can't run and wait. The emulated threads and synchronization primitives which are built on top of these cannot work properly.

There is also PPC CPU emulation discrepancies between Windows and Linux. The registers used on 64-bit Windows and 64-bit Linux are not the same, therefore proper care must be made to use the correct registers when transitioning from guest to host code and vice-versa.

Memory mapping needs a bit of work due to the differences in shared memory and explicit memory ranges having different parameters and prefixes on Linux.

The GTK windowing and its interaction with Vulkan needs to fixed for several bugs.

The use of paths and the string functions need to better specify NT-style line endings (\) which are used by guest and Windows host. In the case of a Linux host, Unix line endings must be used when specifying guest paths.

Linux debugging and stack walking functions need to be implemented.

Approach to Adding Support

My approach to fixing the following problem has been to first add unit tests (and to use the ones already there) to the already working Windows implementation in order to have a behavioural ground truth. Then I add implementation to Linux which satisfy these tests. Once the Linux implementation pass the tests, I activate those tests on Travis in order to prevent future regressions.

Active PRs

Here are the PRs that I have worked on to have native Linux working. Some of these are built on top of the work of others and I preserved the authorship in the commits.
They are in descending order of importance.

Things left to do

If anyone feels like contributing to the port there are a few areas which have not yet been addressed:

Two threading deadlock issues
Vulkan Graphics
Auto Reset events do not work as expected
There is a bug in Event to be fixed

New State

The fixes allow for all unit tests in the project to run and pass on Linux with clang.

The Xenia UI works and can select files with the open dialog.

Loading a xex will work up until emulation.

Calling a host function works as well as host calling a guest function.

Logging works.

Loading a ROM will eventually crash during emulation.

To try all the changes together:

I maintain a branch on my repo: https://github.com/bwrsandman/xenia/tree/linux
If you are running Arch Linux, the folllowing AUR package installs it for you: https://aur.archlinux.org/packages/xenia-git/
You can merge the changes yourself and fix any conflicts:

$ git remote add bwrsandman git@github.com:bwrsandman/xenia.git
$ git fetch bwrsandman
$ git merge                       \
    bwrsandman/linux_cpu          \
    bwrsandman/linux_windowing    \
    bwrsandman/linux_filesystem   \
    bwrsandman/linux_stack_walker

The text was updated successfully, but these errors were encountered:

Prism019 · 2019-08-31T01:12:29Z

linux_stack_walker has a merge conflict in .gdbinit... should probably use -X theirs for that merge?

bwrsandman · 2019-08-31T07:34:49Z

The .gdbinit conflict has to be resolved without an -X strategy.
Unfortunately, using -X theirs will replace the changes from linux_threading and we need both changes from linux_threading (gdb ignoring custom signals from the threading implementation) and those from linux_stack_walker (gdb ignoring custom signal for stack retrieval).
The correct strategy would be -X union but git doesn't currently support that for merge yet.

You can add this to the commands to auto-resolve in a union-like way:

$ sed -i '/^<<<<<<</d;/^=======/d;/^>>>>>>>/d' .gdbinit
$ git add .gdbinit
$ git commit --no-edit

bwrsandman · 2019-08-31T07:47:18Z

Oh yeah, I should add: Use gdb for debugging, lldb is painfully slow and I'm not sure why.

JoelLinn · 2019-10-13T23:37:42Z

What is the current state of these merge requests? There seem to be only minor objections from the maintainers. This work is the basis for all further Linux development.

Prism019 · 2019-10-14T05:39:27Z

Games still do not run. There are also some threading issues (race conditions?) with exiting the emulator. There's also the case of wide chars being different sizes between the guest and host (16 bit vs 32 bit, respectively).

bwrsandman · 2019-10-14T07:05:01Z

I wasn't aware of race conditions...

JoelLinn · 2019-10-14T11:22:00Z

To my knowledge merging this does not break anything, it only improves things.
It would make it easier for others to submit further fixes for linux - doing so right now would probably make them dependent on these PRs anyways.

JoelLinn · 2019-10-15T21:24:15Z

Audio

A cross platform audio back-end is implemented in PR #1498.
It is however untested on Linux (since emulation crashes at the time of writing).
It was way easier than expected to implement this.

This comment was edited since my previous thoughts are no longer relevant

JoelLinn · 2019-10-20T23:42:55Z

Input

Input has been implemented in PR #1493

bwrsandman · 2019-10-21T06:53:46Z

Great work!

bwrsandman · 2019-11-07T09:36:21Z

The window title issue was fixed 0fdbcde
It wasn't related to the different width of wchar. Xenia gets the title correctly from the game file. It was just the printf token which was using regular strings instead of wide strings.

JoelLinn · 2019-11-07T10:49:00Z

Is this it?
I would argue there must be a number of places where sizeof(wchar_t) == 2 is assumed.
E.g. XBox buffers converted to c++ (w)strings and vice versa.

bwrsandman · 2019-11-07T10:53:26Z

Perhaps in src/xenia/kernel/util/shim_utils.h but from my tests running this app, it's only ansi strings being called.

JoelLinn · 2019-11-07T11:57:19Z

Ok a quick static analysis on which code segments assume wchar_t on the host and uint16_t of the guest to be equal:
(May not be complete)

xenia/src/xenia/base/memory.h

Lines 283 to 295 in d4a5a57

    
           template <> 
        
           inline std::wstring load_and_swap<std::wstring>(const void* mem) { 
        
             std::wstring value; 
        
             for (int i = 0;; ++i) { 
        
               auto c = 
        
                   xe::load_and_swap<uint16_t>(reinterpret_cast<const uint16_t*>(mem) + i); 
        
               if (!c) { 
        
                 break; 
        
               } 
        
               value.push_back(static_cast<wchar_t>(c)); 
        
             } 
        
             return value; 
        
           }

xenia/src/xenia/base/memory.h

Lines 402 to 408 in d4a5a57

    
           template <> 
        
           inline void store_and_swap<std::wstring>(void* mem, const std::wstring& value) { 
        
             for (auto i = 0; i < value.size(); ++i) { 
        
               xe::store_and_swap<uint16_t>(reinterpret_cast<uint16_t*>(mem) + i, 
        
                                            value[i]); 
        
             } 
        
           }

xenia/src/xenia/kernel/util/shim_utils.h

Lines 104 to 122 in d4a5a57

    
           inline std::wstring TranslateUnicodeString( 
        
               const Memory* memory, const X_UNICODE_STRING* unicode_string) { 
        
             if (!unicode_string) { 
        
               return L""; 
        
             } 
        
             uint16_t length = unicode_string->length; 
        
             if (!length) { 
        
               return L""; 
        
             } 
        
             const xe::be<uint16_t>* guest_string = 
        
                 memory->TranslateVirtual<const xe::be<uint16_t>*>( 
        
                     unicode_string->pointer); 
        
             std::wstring translated_string; 
        
             translated_string.reserve(length); 
        
             for (uint16_t i = 0; i < length; ++i) { 
        
               translated_string += wchar_t(uint16_t(guest_string[i])); 
        
             } 
        
             return translated_string; 
        
           }

What about using

std::wstring_convert<std::codecvt_utf16<char32_t>, char32_t>

in the case of 4 byte wchar_t platforms to translate between host and guest?

Another way would be to use std::u16string all around - That would however introduce other problems I guess.

bwrsandman · 2019-11-07T14:30:26Z

I went and ran with std::u16string everywhere. No success. It seems to crash at the same place.
0eba293

Prism019 · 2019-11-07T15:09:32Z

That happened to me too on my local branch when i did the conversion. Trying to debug this in GDB is a nightmare though.. when I try to run dolphin.xex, the stack is clobbered when it crashes, so I don't know what to look at in IDA to try and figure out the crash.

JoelLinn · 2019-11-07T15:21:07Z

I went and ran with std::u16string everywhere. No success. It seems to crash at the same place.
0eba293

Eww...
Makes me wonder what the places are where std:wstrings interact with Linux APIs
Maybe use 4 byte in xenia and convert to 1 byte UTF8 when needed.
We would need to do that anyways when using 2 byte strings in xenia.
Downside is one more conversion between xenia and guest (4 byte <> 2 byte).

Generally speaking, a clean solution on Linux is to use std::string with UTF8 encoding but that gets pretty messy when you have ansistrings around that are also 1 byte long like we have.

Prism019 · 2019-12-02T20:33:05Z

For anyone not following along on the #dev-linux channel on the discord, I've gotten the game emulation to be stable. I'm going to check to make sure the fixes don't interfere with Windows, then open a new PR to get the changes merged.

Note: Vulkan Graphics and SDL Audio do not work as of now. SDL Input may work but I have no way of testing it in a game.

Prism019 · 2019-12-03T17:37:22Z

#1517 should fix the segfaulting while running emulations.

bwrsandman · 2019-12-04T09:51:37Z

There are some conflicts with #1339 I don't know if you tested with both, I'm applying #1517 after #1339 with the -X theirs strategy.
What are your thoughts?

bwrsandman · 2019-12-07T20:46:11Z

There seems to be a bug in the auto reset events in the threading PR.
The notify one signal does not seem to unblock the main thread causing a deadlock.
More tests need to be made to have the behaviour on par with windows.
See the threading PR for more details.

sl1pkn07 · 2020-03-02T18:48:28Z

seems conflict #1339 with #1397 and #1405 with #1317

greetings

bwrsandman · 2020-03-03T19:34:20Z

Rebased and fixed the conflicts.
I did minimal testing as I don't have a lot of time to dedicate to this.

Shoegzer · 2021-05-04T16:45:45Z

Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d9.

I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date.

Thoughts?

SerraraFluttershy · 2021-06-02T09:05:39Z

Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d9.

I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date.

Thoughts?

I concur with this comment. As a frequent user of this software (for playing games and high-level reverse engineering of said games) I also recommend all non-problematic pending commits be merged into the master branch so that more pressing matters (such as Vulkan rendering and windowing) can be addressed on the journey to a fully functional Linux release.

Should I try compiling a custom build of the software myself (incorporating the pending commits of course) and relay the results of doing so back to you all?

JoelLinn · 2021-06-02T09:30:14Z

Thanks to PR #1792, Xenia finally builds on a local linux test system. While that's great progress, the resulting (debug) build predictably crashes, as mentioned above and shown here. Tested on Linux Mint 20.1 / kernel 5.11 with commit 95031d9.
I can do some testing with this PR to help out, but agree with @JoelLinn that it may be best to merge as-is, especially given the lack of activity in over a year at this point. Now that Linux builds are compiling in master, I can imagine devs and testers alike could produce more useful results without a need to be aware of or dependent upon this PR. I can only assume the checkoff lists mentioned in the summary are still up-to-date.
Thoughts?

I concur with this comment. As a frequent user of this software (for playing games and high-level reverse engineering of said games) I also recommend all non-problematic pending commits be merged into the master branch so that more pressing matters (such as Vulkan rendering and windowing) can be addressed on the journey to a fully functional Linux release.

Should I try compiling a custom build of the software myself (incorporating the pending commits of course) and relay the results of doing so back to you all?

I think biggest hesitation for merging is with #1339 I think. It modifies code that is shared among platforms and has some open discussions.
IIrc stuff like windowing changes are independent to CPU

sl1pkn07 · 2021-08-08T16:27:49Z

Hi

for me, without any patch or applied any PR, fail build in my archlinux (own package)

./src/xenia/base/filesystem_posix.cc:48:3: error: ignoring return value of function declared with 'warn_unused_result' attribute [-Werror,-Wunused-result]
  readlink("/proc/self/exe", buff, FILENAME_MAX);
  ^~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:268: obj/Linux/Release/Linux/Release/xenia-base/filesystem_posix.o] Error 1
make[1]: *** Se espera a que terminen otras tareas....
../src/xenia/base/system_linux.cc:22:10: fatal error: 'third_party/SDL2/include/SDL.h' file not found
#include "third_party/SDL2/include/SDL.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:307: obj/Linux/Release/Linux/Release/xenia-base/system_linux.o] Error 1
../src/xenia/base/memory_posix.cc:136:3: error: ignoring return value of function declared with 'warn_unused_result' attribute [-Werror,-Wunused-result]
  ftruncate64(ret, length);
  ^~~~~~~~~~~ ~~~~~~~~~~~
1 error generated.
make[1]: *** [xenia-base.make:289: obj/Linux/Release/Linux/Release/xenia-base/memory_posix.o] Error 1
make: *** [Makefile:290: xenia-base] Error 2
make: se sale del directorio '/tmp/makepkg/sl1-xenia-git/src/xenia/build'

ERROR: build failed with one or more errors.

commit used: 4861022

clang 12.0.1
kernel 5.13.8-arch1-1

#1339, , #1405, #1431, #1433 needs rebase due conflict files

EDIT: fixed with

diff --git a/premake5.lua b/premake5.lua
index e44f81c71..463264402 100644
--- a/premake5.lua
+++ b/premake5.lua
@@ -131,7 +131,8 @@ filter({"platforms:Linux", "toolset:gcc"})
 
 filter({"platforms:Linux", "language:C++", "toolset:clang"})
   disablewarnings({
-    "deprecated-register"
+    "deprecated-register",
+    "unused-result"
   })
 filter({"platforms:Linux", "language:C++", "toolset:clang", "files:*.cc or *.cpp"})
   buildoptions({

greetings

sl1pkn07 · 2021-08-09T16:57:12Z

seems #1076 needs to be added to the list. with this PR now i can able to run xenia without segfaults and initated properly the vulkan subsystem on my linux

greetings

Triang3l · 2021-08-09T17:00:01Z

Yes, #1076 is an even more advanced and comprehensive version of #1431 conceptually.

CtrlC-Root · 2023-05-31T19:35:21Z

#2018 seems to supersede #1339

xiota · 2023-07-07T19:24:23Z

Need to add #include <cstdint> to

xenia/src/xenia/base/utf8.cc

Lines 12 to 16 in 00aba94

    
           #include <algorithm> 
        
           #include <locale> 
        
           #include <numeric> 
        
           #include <tuple>

~~Or change uint32_t to std::uint32_t ?~~

illusion0001 · 2023-07-07T20:47:04Z

Fixed in #2211 I believe

Velkee · 2023-09-15T19:12:40Z

Hi! From what I can gather from the thread Linux support is still very experimental and broken, but I want to report some issues I'm having while attempting to build the AUR package.

There seems to be several errors happening while trying to compile libzip for building premake5.

Error one appears as follows:

../../contrib/libzip/mkstemp.c:76:8: error: call to undeclared function 'getpid'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        pid = getpid();
              ^

Error two states this:

../../contrib/libzip/zip_close.c:621:2: error: call to undeclared function 'close'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
       close(tfd);
       ^

If this is completely unrelated then please excuse me.

sebastien46 · 2023-10-05T19:31:43Z

Hi! From what I can gather from the thread Linux support is still very experimental and broken, but I want to report some issues I'm having while attempting to build the AUR package.

There seems to be several errors happening while trying to compile libzip for building premake5.

Error one appears as follows:
../../contrib/libzip/mkstemp.c:76:8: error: call to undeclared function 'getpid'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        pid = getpid();
              ^
Error two states this:
../../contrib/libzip/zip_close.c:621:2: error: call to undeclared function 'close'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
       close(tfd);
       ^
If this is completely unrelated then please excuse me.

When trying to build from AUR this error occurs for me too.
From reading the log, I think this error occurs because premake-core uses a very old version of libzip from 2015 (0.11.2 to be precise), so it needs to be updated to get past this error (see #2121).

bwrsandman · 2023-11-19T20:53:46Z

AUR issue has been resolved.
There's a needed patch for the POSIX console main to remove the use of the logger to fix a linker error. That should be looked into.

This was referenced Aug 14, 2019

Linux windowing fixes #1431

Merged

Linux filesystem #1433

Open

Linux misc fixes #1322

Closed

Margen67 added the platform-linux label Aug 17, 2019

oscarbg mentioned this issue Aug 24, 2019

Expose primitive ordered pixel shaders GPUOpen-Drivers/AMDVLK#108

Closed

JoelLinn mentioned this issue Oct 20, 2019

[HID/Linux] Support Linux systems and non XInput controllers #1493

Merged

bwrsandman mentioned this issue Mar 3, 2020

[cpu linux] Implement stack walking #1405

Open

bwrsandman mentioned this issue Mar 3, 2020

Linux memory Fixes #1397

Merged

sharkwouter mentioned this issue Apr 9, 2020

RPCS3 Support ChimeraOS/chimera#63

Open

Triang3l mentioned this issue Jul 26, 2020

[REQUEST] Support Linux nativly #1633

Closed

JoelLinn mentioned this issue Jan 10, 2021

Linux compile crash with errors #1735

Closed

JoelLinn mentioned this issue May 24, 2021

Build error on gentoo #1819

Closed

JoelLinn mentioned this issue Jan 19, 2022

Build error in Ubuntu #1958

Closed

JoelLinn mentioned this issue Feb 21, 2022

Fix posix threading stuff 😿 #1992

Merged

iMacker2020 mentioned this issue Mar 29, 2022

Xenia for Mac? #596

Open

Linux Support #1430

Linux Support #1430

Comments

bwrsandman commented Aug 14, 2019 • edited by Triang3l

Current State

Approach to Adding Support

Active PRs

Things left to do

New State

Prism019 commented Aug 31, 2019

bwrsandman commented Aug 31, 2019 • edited

bwrsandman commented Aug 31, 2019 • edited

JoelLinn commented Oct 13, 2019

Prism019 commented Oct 14, 2019

bwrsandman commented Oct 14, 2019

JoelLinn commented Oct 14, 2019

JoelLinn commented Oct 15, 2019 • edited

Audio

JoelLinn commented Oct 20, 2019

Input

bwrsandman commented Oct 21, 2019

bwrsandman commented Nov 7, 2019

JoelLinn commented Nov 7, 2019

bwrsandman commented Nov 7, 2019 • edited

JoelLinn commented Nov 7, 2019

bwrsandman commented Nov 7, 2019

Prism019 commented Nov 7, 2019

JoelLinn commented Nov 7, 2019

Prism019 commented Dec 2, 2019 • edited

Prism019 commented Dec 3, 2019

bwrsandman commented Dec 4, 2019 • edited

bwrsandman commented Dec 7, 2019

sl1pkn07 commented Mar 2, 2020 • edited

bwrsandman commented Mar 3, 2020

Shoegzer commented May 4, 2021

SerraraFluttershy commented Jun 2, 2021

JoelLinn commented Jun 2, 2021

sl1pkn07 commented Aug 8, 2021 • edited

sl1pkn07 commented Aug 9, 2021 • edited

Triang3l commented Aug 9, 2021

CtrlC-Root commented May 31, 2023

xiota commented Jul 7, 2023 • edited

illusion0001 commented Jul 7, 2023

Velkee commented Sep 15, 2023

sebastien46 commented Oct 5, 2023 • edited

bwrsandman commented Nov 19, 2023

bwrsandman commented Aug 14, 2019 •

edited by Triang3l

bwrsandman commented Aug 31, 2019 •

edited

bwrsandman commented Aug 31, 2019 •

edited

JoelLinn commented Oct 15, 2019 •

edited

bwrsandman commented Nov 7, 2019 •

edited

Prism019 commented Dec 2, 2019 •

edited

bwrsandman commented Dec 4, 2019 •

edited

sl1pkn07 commented Mar 2, 2020 •

edited

sl1pkn07 commented Aug 8, 2021 •

edited

sl1pkn07 commented Aug 9, 2021 •

edited

xiota commented Jul 7, 2023 •

edited

sebastien46 commented Oct 5, 2023 •

edited