Big endian

[pbekesky]

Hi

I wanted to ask if it has been tested/is it running on big endian systems?
I was able to recompile the project on big endian system, after starting it up it loads all the files, opens main window, music starts to play and then it crashed in memory exception.
I found that is looping endlessly in CelDrawDatOnly() method in engine.cpp until it finally ends on line 36:

v7 = (unsigned char)*v4++;

I'm trying to understand what is going on here and how it should work here, but I'm unable to get it working. Stormlib is prepared for big endian and it seems to working fine but it seems to bes something wrong in the CelDrawDatOnly() function
Is any of the developers willing to help with that? I can provide more info if needed.
Thanks

[AJenbo]

engine.cpp is in a pretty poor state, there are still a lot of it that dosen't even work on 64bit. The original code was ported to bi endian so this is some thing that we would want to do also. Both asset loading and savegames where compatible between i386 and powerpc.

What CPU architecture are you specifically working with?

[pbekesky]

CPU: Freescale P5020 CPU 2GHz, 64-bit e5500 dual-core PowerPC SoC
OS: AmigaOS 4.1

CPU has 64 architecture but OS is only 32bit with single core support at the moment. 64bit/multicore in OS is under development.
More info:
https://www.amigaos.net/hardware/133/amigaone-x5000
Or just feel free to ask :)

[MBeijer]

I am eagerly awaiting progress on this as well for m68k AmigaOS 3.x. ^^

…

On Wed, Feb 13, 2019 at 11:49 AM infinityx ***@***.***> wrote: CPU: Freescale P5020 CPU 2GHz, 64-bit e5500 dual-core PowerPC SoC OS: AmigaOS 4.1 CPU has 64 architecture but OS is only 32bit with single core support at the moment. 64bit/multicore in OS is under development. More info: https://www.amigaos.net/hardware/133/amigaone-x5000 Or just feel free to ask :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#21 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIXahc2_wKF2vUES_JbxlJNJdeOkRaQRks5vM-3AgaJpZM4a4yM8> .

[AJenbo]

PPC should be feasible, probably 32bit OS is probably a benefit as things currently stand. A sufficiently clocked m68k should work out, but the port at the moment is a bit memory hungry (~300mb vs ~45mb), my biggest concern for a m68k would be the graphics card, it needs to either be able to page flip or blit 640x480 @ 20hz.

@InfinityX I take it you where able to get the menu running? CelDrawDatOnly() shoudn't be invoked before the actual game world loads.

[MBeijer]

I would port it for the Vampire accelerator, which should be beefy enough to handle it. :)

…

On Wed, Feb 13, 2019 at 3:40 PM Anders Jenbo ***@***.***> wrote: PPC should be feasible, probably 32bit OS is probably a benefit as things currently stand. A sufficiently clocked m68k should work out, but the port at the moment is a bit memory hungry (~300mb vs ~45mb), my biggest concern for a m68k would be the graphics card, it needs to either be able to page flip or blit 640x480 @ 20hz. @InfinityX <https://github.com/infinityx> I take it you where able to get the menu running? CelDrawDatOnly() shoudn't be invoked before the actual game world loads. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#21 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIXahZfUv7C7y9A-IU2hWnOlgOZg_T6Cks5vNCP3gaJpZM4a4yM8> .

[pbekesky]

@AJenbo
Actually, whole window is pitch black, just music is playing. But considering Your question, I should look at the menu first. I understood that CelDrawDatOnly() is handling menu drawing, but I'm probably wrong :)

Can You give me some leads where to look at?

[AJenbo]

The menu is all PCX and rewritten since the original is heavily tied to Windows in a way that's not even supported by modern Windows. The way we have rewritten the menu it is now a part of the main application and a bit more tightly coupled to the rest of the engine. This should make it more portable.

I'm currently working on refactoring the menu code as the current one isn't really nice to work with, i have pushed the rewrite to the menu branch so you can start by looking at it.

cel and cl2 are only used in game, including the in game menu, meaning that CelDrawDatOnly() isn't invoked before you create your hero and start a game.

In regards to porting to big endian I think engine.cpp is the main place this would need to be done for. pfile.cpp would need similar treatment to have save games be compatible.

You will fine the menu code in Stubs/DiabloUI (mainly diabloui.cpp), blitting happens in Stubs/dx.cpp BltFast(), SDL render initialization happens in Stubs/init.cpp init_create_window()

Make sure to join us on Discord as that's a much easier way to talk how things work :)

[grepwood]

I could test this on PS3. Both GameOS and Gentoo Linux.

[grepwood]

Some food for thought. Regarding coping with alien-endian data.

Sadly each compiler uses different intrinsics for byteswapping. But here's how I wrapped up this mess in FreeRadical:

#ifdef __APPLE__
#       include <libkern/OSByteOrder.h>
#       define FR_bswap16(x) OSSwapConstInt16(x)
#       define FR_bswap32(x) OSSwapConstInt32(x)
#       define FR_bswap64(x) OSSwapConstInt64(x)
#elif defined __GNUC__ && defined linux
#       include <byteswap.h>
#       define FR_bswap64(x) __bswap_64(x)
#       define FR_bswap32(x) __bswap_32(x)
#       define FR_bswap16(x) __bswap_16(x)
#elif defined WINNT
#       include <x86intrin.h>
#       define FR_bswap16(x) (uint16_t)(__bswapd((uint32_t)x)>>16) /* This is GCC on Windows, not MSVC */
#       define FR_bswap32(x) __bswapd(x)
#       define FR_bswap64(x) __bswapq(x)
#else
#       error "This compiler is not supported" /* plain C implementations go here */
#endif

These intrinsics will find either backing in hardware or efficient software implementations depending on your target and CFLAGS.

We can take things to the next step and target even larger chunks of alien-endian data with SSSE3 _mm_shuffle_epi8 , AltiVec vec_perm and even AVX2 _mm256_shuffle_epi8 . By using appropriate shuffle masks we can byteswap from 8 or 16 shorts to 1 or 2 long doubles at a time depending on which SIMD is used.

[AJenbo]

Nice, I think this will cover our need for the time being

[AJenbo]

engine.cpp is now in a much better state and I was able to get the game loading all the way into town with UI and simple items like mana potions rendering in the game world as well. The man changes needed can be gleaned from here:
diasurgical/devilution#1253 (comment)

For the game world to render some more work would be needed to render.cpp. Since upstream probably won't do much more work on it (the original being written in ASM, a more viable path would be the cleaned up render found in this branch:
https://github.com/diasurgical/devilutionX/blob/render/Source/render.cpp

[MBeijer]

For anyone following this thread, work is being done for Big endian systems in this fork: https://github.com/AmigaPorts/devilutionX
A proper PR will come when it's nearing 100% working

[AJenbo]

@MBeijer in case you haven't already noticed I just merged an SDL1 implementation done by @glebm, not sure if it will be helpful to your efforts at this point, but maybe you can swap notes and i think it is probably a good starting point for upstraming parts of the port. I have also been working on cleaning up some of the parts of the code that where hard to port, like init, mpq loading, savegames and error handling.

I also opened a draft PR for your branch so that I might better be able to follow along withe the port and possibly implement parts of it during refactoring.

[MBeijer]

lmao, @glebm's SDL1-implementation is essentially the same as ours. :P Just a wrapper for SDL2 specifics and ifdef's. The difference is that we use #if SDL_VERSION_ATLEAST(2, 0, 0) to check if SDL2 is in use. Glebm's is less feature complete than ours, but his wrapper file is way cleaner. :P

[MBeijer]

Ours also support audio, except for the intro.

[MBeijer]

Ping @arczi84

[arczi84]

And fullscreen works.

[glebm]

I hadn't realized there was already an SDL1 implementation out there. That would've saved me some time but I hadn't though anyone else was crazy enough to try writing one 😂

I've just sent a PR with SDL1 audio support for movies btw

Curious how you got fullscreen working, can you please upstream it?
Nothing I tried works, but I have a weird multi-monitor setup and maybe that's confusing SDL.

[arczi84]

Look here https://github.com/AmigaPorts/devilutionX/tree/master/

[AJenbo]

Or here https://github.com/diasurgical/devilutionX/pull/268/files

lmao, @glebm's SDL1-implementation is essentially the same as ours. :P Just a wrapper for SDL2 specifics and ifdef's. The difference is that we use #if SDL_VERSION_ATLEAST(2, 0, 0) to check if SDL2 is in use. Glebm's is less feature complete than ours, but his wrapper file is way cleaner. :P

Yeah that is also the impression I got, though I haven't gone over your in detail yet. At first I actually though it was the one from your branch. So I feel this is a decent way to go in terms of up streaming and luckily it happened at a point where you both have some different features that can be combined :)

[AJenbo]

The custom wave loading has now been dropped and SDL is used instead, so audio should now work on big endian systems.

[arczi84]

It has already been working, except in movies was just noise. śr., 25 wrz 2019, 11:37 użytkownik Anders Jenbo <notifications@github.com> napisał:

…

The custom wave loading has now been dropped and SDL is used instead, so audio should now work on big endian systems. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21?email_source=notifications&email_token=AA2X25TOARXILNGGLQCHWO3QLMWMFA5CNFSM4GXDEM6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RIPFI#issuecomment-534939541>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2X25TVOVXSI4FYVPVUMMLQLMWMFANCNFSM4GXDEM6A> .

[AJenbo]

Not in the main branch afaik

[AJenbo]

The movie audio is also fixed now 3b2ba95

[AJenbo]

You can now walk around town if omitting pfile_read_player_from_save d641c65

[AJenbo]

Cathedral is now playable, the game will crash on some level loads, likely because of loading .dun maps. 1ab2fb2

[AJenbo]

#548 takes care of save games
29881d2 looks to have fixed issues with loading quest levels

Having playtested I see no issues with the game on big-endian at this point.