SMBX memory emulator planning

[Wohlstand]

Before start backporting the features of LunaDLL (and then LunaLua), we should build the memory emulator that will hold all known and used in-SMBX memory addresses that leads to various fields, variables, and other stuff that used the direct memory hacking in the SMBX. There are some quite from the Discord chat where the idea was been discussed.

WohlstandFox
Anyway, once I'll make some sort of the memory emulator, will be possible to play some of the pre-SMBX2 episodes with scripts on here. That the first stage.

Nomagno
do you think they'd be willing to release the source of their memory hacks?
if it's VB6 we can probs do something about it

WohlstandFox
There are already LunaLua itself is open-source and the repo is in public, also some of Rednaxela's pages with the memory tables are here

Nomagno
I mean SMBX2
as a whole

WohlstandFox
at PGE Wiki there are a lot of tables written against the known memory addresses

WohlstandFox
The rest of these addresses I'll be able to emulate directly by using my smbx-experiments libsmbx' experiments
Anyway, there are a lot of scripts that mention addresses, and the IDA where are most of the known variables were marked
I do have a big archive of information about know memory addresses that I will be able to use in my memory emulator
what the memory emulator will do? simply a huge switch-case where all known addresses mentioned, and the input/output conversion for the every value type
I'll try some on days off, but NO HURRY, mainly because I do have the active shit at the main job I must to kill finish
Then, if successful, I'll have a two-week vacation
where I will have much more free time to work on projects
Even when I'll visit the village eventually, I have a laptop that I can take with me and keep work on the project

The most of extra stuff was written in the lua, and also there is LuaJit's FFI technology got been utilized to improve the performance while exchanging the data between C++ and Lua engines
btw, @Rednaxela#0380, a stupid question: if I do have at me the array of int while in the past it was an array of short, will that hurt the lua part when I'll change the short with int? In a sense, if pass the array by FFI

Rednaxela
I mean, the Lua side would require corresponding updates to deal with an int array instead of short array, so yes, unless said changes were made

WohlstandFox
Mainly because all Integer (equivalent of 16-bit short) types I replaced with 32-bit int (that automatically resolved a lot of oveflow crashes and array indeces messing up), but the code in the whole works absolutely equal to the original VB6 include the most of bugs, except cases I messed up by myself that was been fixed later

WohlstandFox
the binding code you mean?
(that makes the referece of the native array at lua side directly)

Rednaxela
any code that interacts with such via FFI would need to be updated, generally meaning the binding code in Lua yes

WohlstandFox
but will that harm the other lua code that simply uses it?

Rednaxela
I'm not quite sure what other code using it you mean or how it would interact

WohlstandFox
Anyway, I think, it shouldn't harm any lua code that has access to these arrays by mentioning of the binding
(the binding itself must match to the actual native code, that is yes)

Rednaxela
as long as the binding itself is updated

WohlstandFox
yes
By the fact, while making the memory emulator, I will keep the same addresses of old existing fields, but new-added fields will be not accessible by mem(). Btw, another question: are there codes that doing the memory block operations than just value read/write?
I mean such of memcpy / memset, etc.
which will affect the whole range than just the every field

Rednaxela
"are there codes" where?

WohlstandFox
at LunaLua and SMBX2 I mean

Rednaxela
It's uncommon, but there are rare instances of code doing bulk copies of data structures via mem calls, with special exceptions made only for string pointers
not in anything in lunalua/basegame smbx2, but some scripts such as blockcleanser and such

BlockCleanser.lua.txt

Rednaxela
basically a little trickery to have blocks unloaded when not in the active section
note that as implemented it's not safe to use in many levels, many corner cases it doesn't handle, but it handles what was needed for the maglx3 hub sufficiently

WohlstandFox
I see, and, in most, the code do use the per-value mem() calls 👀
just the tricky code to simulate the behavior
Also, at TheXTech all strings are std::string, so, the VB6Ptr will be a fake overlay for the backward compatibility

Rednaxela
The notable thing for what you're thinking about doing, is how this code uses FIELD_DWORD regardless of type (except for strings), and the FIELD_DWORDs it saves can overlap multiple fields (if shorts/bools), or split fields (if doubles)

WohlstandFox
I'll simply trick on the value expected and will try to simulate the same behavior as is actual, that will also need a unit test

so, bitwise operations will be uses to trick middle or incorrect addresses

int field1;
int field2;
uint32_t dword;

// 0 1 2 3
field1 = (dword & 0xFFFF);
field2 = (dword & 0xFFFF0000) >> 16;

🤔

also, knowing about alignment gaps is possible to ignore some, or... preserve to return back
(by the fact, these alignment gaps may be tricked and used as values, technically)
And backward:

int field1;
int field2;
uint32_t dword;

// 0 1 2 3
dword = (field1 & 0xFFFF) | ((field2 & 0xFFFF) << 16);

Rednaxela
Mhmm

WohlstandFox
so, as the memory emulator hosts the emulation of SMBX's align gaps, I'll store at the memory emulator directly by using maxBlocks / maxBGO / etc. to define the array size. These values must not be touched by the engine, as at me there are Block_t()-like constructors being used to resetting all entry fields into the default
and storing gaps at structures will also reset them too on re-construction
And yeah, writing of double fields should be cast into byte arrays then (Usually I do that to convert their endianness at my another project)
I guess, to simplify the logic the memory emulator will hold the sort of "map" with the size of every field and their real types where it will be able to trick them easily
(where the real type from SMBX's is different, do conversion here)

Rednaxela
(if you really want to be cross-platform when casting double fields to bytes, you should be aware that some versions of ARM use an atypical encoding for doubles... each 32-bit half is little-endian, but the two 32-bit halves are stored in big-endian order)

WohlstandFox
I see, then define the processor-depend trick logic (I'll need to make the unit test to print these doubles and compare with their x86 siblings), very interesting, thanks for the hint :fox:

WohlstandFox
My goal is simply to allow DWORD/WORD to be written over double fields and keeping the same result on both sides, therefore I should check out what happens with double values when they got written by such stuff (in some cases they would just get the NaN as the damaged dot position code)

Rednaxela
Mhmm

WohlstandFox
Also, I guess, the same may happen if you do send the double by the network between x86 and ARM machines
https://stackoverflow.com/questions/57402613/serializing-a-double-for-transmit-over-udp-arm-to-x86 🤔
Anyway, I'll need that, mainly to take from memory emulator in x86 format, and then convert into ARM format when needed

Rednaxela
Yes. Properly designed binary network protocols always need to specify data types more specifically than just giving a C data type. Always need to be clear about endianness

WohlstandFox
Yeah, or encode into a special hand-made format that is safe to decode on the target device (at my job we got one device that had the 3-byte-encoded floating-point number that uses a tricky formula to encode and decode)

Rednaxela
(one thing that's funny is most traditional network protocols standardized on big-endian, even though most popular processors these days are natively little-endian)

WohlstandFox
PDP and PowerPC era

[LoveBodhi]

That's system for SMBX2. Maybe there're script support?

[Wohlstand]

Also for LunaDLL AutoCode (the simple script language used before lua was added). At the first, I gonna make pre-SMBX episodes with scripts to work here. But the fact direct-memory hacking calls existed, I must emulate this behavior for backward compatibility. Technically, SMBX2's "empire" can be ported too, but that will need some time a bit later. No hurry, do the work gradually step-by-step to get the finish eventually than stuck in the chaos.

[LoveBodhi]

Therefore, when this done, original FTW "Standard Edition" will rename to "Lite Edition", the new "Standard Edition" will include enhanced features as well (e.g. Parallax BG's).

[Wohlstand]

The explanation on how double floating point type works on some ARM processors:
https://stackoverflow.com/questions/10142684/wrong-conversion-from-double-to-qstring-in-qt-on-arm
They do have the mixed order:

11 22 33 44 55 66 77 88 - little endian
88 77 66 55 44 33 22 11 - big endian
55 66 77 88 11 22 33 44 - mixed endian

---

A useful link given by @Bluenaxela:

https://stackoverflow.com/questions/26105686/correctly-detect-mixed-endian-floating-point-format-on-arm-platform

This should help

[Wohlstand]

While I helped @suicvne to fix the "toNearest()" on Vita, I made some call that should help with ARM-related things:

#if defined(arm) && !defined(__SOFTFP__) && !defined(__VFP_FP__) && !defined(__MAVERICK__)
inline void swap_halfes(uint64_t &x)
{
    uint64_t y = ((x & 0xFFFFFFFF00000000) >> 32) & 0xFFFFFFFF;
    x = ((x << 32) & 0xFFFFFFFF00000000) & y;
}
#else // Do nothing
#   define swap_halfes(x)
#endif

I'll use this at these places that require the halves swap when processing the number in its binary form and use it again to get back the native form.

Does it looks fine? @Bluenaxela?