[rREM] Update raylib Resource EMbedder!

[raysan5]

Update raylib Resource Embedder (rREM)...

[raysan5]

Just started work on the new rRES (raylib resource) custom file format. Here a pre-liminary version of the API: rres.h

[raysan5]

RRES file format support will be ready for raylib v1.7 but visual tool could take a bit longer. Probably rREM will be command line only on first implementation...

[gen2brain]

Hi Ray,

I now you are busy with new release, just wanted to ask what is the status of rREM util? In Go bindings I started with something like this https://github.com/gen2brain/raylib-go/blob/master/raylib/rres.go , but I can't test, source for rREM util is missing, and I am not sure what final format would be.

I would like to make a binary util for that, in native Go it can be easily cross compiled.

[raysan5]

Hi Milan!

Lately I've been pretty busy in my daily job, preparing classes, setting and correcting exercises, preparing exams... I didn't have much time to focus on rres.h (implementation isn't complete), neither in rREM; latest implementation of that tool dates from 2014 (completely outdated, needs to be reworked from scratch) and the format has been improved a lot... despite it still requires some decisions:

• Resource part parameter is used as counter for resources divided in multiple parts, is it the best way to consider that situation?
• Add at the end of the file a central directory with original filenames and offsets within the file (similar to .zip fileformat), should it be added?
• I've studied the RIFF fileformat, don't like the tree-style implementation, I prefer to keep it simple, just one chunck after another but I want to study more deeper the pros vs const.
• Most of the functionality of rREM is being moved directly to rres.h, rREM is becoming just a front-end and command-line tool using rres.h.

In any case, I attach old rREM implementation for reference (rREM.zip) but this tool is deprecated right now...

Ok, my plan, I'll publish raylib v1.7 as soon as possible (no more than two weeks, despite not being complete with all the features I wanted...) and I'll put on high priority this tool.

[gen2brain]

I implemented resource embedder in Go based on what is currently in rres.h https://github.com/gen2brain/raylib-go/tree/master/rres/cmd/rrem . It is in native Go so it is easy to compile or cross compile (just go build, no need for C compiler etc.) , binary format should be compatible.

There are also options to generate .h and .c files, and it can generate .go file so everything can be compiled in binary, encrypted and compressed. Currently I have an issue with deflate (only for images, wav's are ok, they are incomplete, not sure if that is only read or both read/write issue, it happens with both official and the library I use) so default is LZMA2 (XZ).

• It is usable only with images, wav and text/raw, currently there are no Ex functions to load .ogg for example.
• Not sure how PartsCount is supposed to work so for now there is only 1 part.
• Still doesn't embed vertex and font data, I found lib that can parse .obj files though

Here is the Windows binary in case you want to check, http://81.4.106.254/rrem.exe . Maybe there are no issues with C deflate.

[raysan5]

Hi @gen2brain! Great work!

Sorry for forgetting about this issue... I've been busy with lots of things.

Not sure how PartsCount is supposed to work so for now there is only 1 part.

I think it's not clear enough for anyone... The objective was using it to define differents chuncks of data (parts) for a related piece of data... for example, SpriteFonts, two parts, one part is the sprite image atlas and second part is the charset info data. Another example, Models, they are composed of multiple vertex data chuncks of different type. But managing those chunks of data on loading is a bit complicated...

I tried to download the Windows binary but my system blocks the file, could you rename it to something like rrem.renamed and share again?

[gen2brain]

Hi, file is renamed, http://81.4.106.254/rrem.renamed . Also, I opened issue about DEFLATE here klauspost/compress#82 . Not sure what I am doing wrong there, because it works with other compressed formats.

Btw. it is posible to create a library in Go that only has Compress/Decompress and Encrypt/Decrypt functions, with all algorithm supported, i.e. DEFLATE, LZ4, LZMA2, BZIP2, AES, 3DES, Blowfish, XTEA etc. and then compile to C static library with go build -buildmode=c-archive , result is static .a library with header .h file, so that can be used in C.

It maybe sounds strange, but then you would not need all the dependencies for C version.

[raysan5]

Hey @gen2brain! Just tested it! It works great! :D

About DEFLATE, why not using miniz, it worked great for me...

Btw. it is posible to create a library in Go that only has Compress/Decompress and Encrypt/Decrypt functions, with all algorithm supported, i.e. DEFLATE, LZ4, LZMA2, BZIP2, AES, 3DES, Blowfish, XTEA etc. and then compile to C static library with go build -buildmode=c-archive , result is static .a library with header .h file, so that can be used in C.

Wow! That would be amazing! Which libraries are you using for all those algorythms? I understand that generated .a library has no external dependencies at all, right?

It maybe sounds strange, but then you would not need all the dependencies for C version.

Afte some years in coding, what would sound strange to me would be something not being possible to do... :P

[gen2brain]

DEFLATE issue was my fault, it is fixed now.

Most of the libraries are from official Go standard lib, like https://golang.org/pkg/crypto/#pkg-subdirectories and https://golang.org/pkg/compress/ , maintained and well tested code. Some are from https://github.com/dsnet/compress and https://github.com/klauspost/compress , some of that work is probably going to be also included in standard lib.

Yes, generated .a will have no external dependencies, I did something similar recently with other project. I will try something these days.

Something like this is ok I guess:
unsigned char *Compress(const unsigned char *data, unsigned long uncompSize, unsigned long *outCompSize)
unsigned char *Decompress(const unsigned char *data, unsigned long compSize, int uncompSize)

[raysan5]

Yes, generated .a will have no external dependencies, I did something similar recently with other project.

That's great!

Something like this is ok I guess...

Absolutely! By the way, how you determine the compression algorythm with those functions? Maybe using:

unsigned char *Compress(int compType, const unsigned char *data, unsigned long uncompSize, unsigned long *outCompSize)

Just an idea...

[gen2brain]

Sure, compType and encType for Encrypt/Decrypt will need to be passed. I forgot that.

[gen2brain]

Ok, I created Cgo library here https://github.com/gen2brain/raylib-go/tree/master/rres/rlib/cgo . Some binary releases are here http://81.4.106.254/rlib-release.zip .

I don't like it, header files are platform dependent and they generate something like this, without parameter names that are defined:
extern unsigned char* Compress(int p0, unsigned char* p1, long unsigned int p2, long unsigned int* p3);

I guess just DEFLATE and XOR will be fine enough. Main feature I see here is to compile resources together with binary, not to compress and encrypt with some ultra encryption so nobody can extract your file.

[raysan5]

Hi @gen2brain, excuse me I'm a bit confused, probably due to naming...

So, rlib is the Compression/Decompression/Encrypt/Decrypt C library automatically generated with cgo... and cgo is a Go tool that enables interconnection between C and Go, is that right?

Maybe naming it rreslib would be more clear to me but don't know if I'm wrong with the assumption, I mean, maybe rlib is a generic data Compression/Decompression/Encrypt/Decrypt library, independent of any data packaging format... in that case, naming could be something else... or just rlib...

[gen2brain]

Yes, naming is bad, should be renamed to rreslib. Yes, Cgo is a tool for bridging C and Go, either to use library from C in Go, or to export/generate library that can be used in C.

How it is currently, here is the lib in Go https://github.com/gen2brain/raylib-go/blob/master/rres/rlib/rlib.go , and this is a wrapper to cgo lib https://github.com/gen2brain/raylib-go/blob/master/rres/rlib/cgo/rlib.go.

[gen2brain]

Renamed here gen2brain/raylib-go@275bcd3 .

[raysan5]

Just moved project to own repo: https://github.com/raysan5/rres

Also worked a bit further on format design, I'll upload a new specs design this weekend... basically, I support multiple data chunks per resource (similar to RIFF file-format).

After thinking quite long on that, I think it's the best option, it also adds a lot of versatility for users to customize own resources data.

[raysan5]

Here it is the proposed update for rRES:

I reviewed the specs based on feedback provided on Handmade Network post and a more deep review of RIFF and ZIP file-formats.

My only concern is, maybe, using on each data chunk 4 byte for chunkType, compType, crypType, paramCount and use 4 bytes for a CRC32, I mean, this:

chunkInfoHeader:

What do you think?

EDIT: After looking at it more carefully, I think the version with CRC32 is better.

[raysan5]

By the way, probably this discussion should be moved to new rres repo: https://github.com/raysan5/rres

[raysan5]

Just removed .rres file support from raylib in commit a6f9cc5.

This way we let the user to choose if using custom resource files, just including the rres.h library.

Additional work has started on new rrem tool based on new rres format specs, along rrem design I will see format requirements and possible changes and improvements.

[raysan5]

By the way, @gen2brain, did you see this issue? what are your thoughts?

[gen2brain]

Looks great, I would prefer to go without that crc32 if possible, only complicates things it seems.

So rres.h is not required anywhere anymore, right? That is great, I would also like to remove that from raylib/ directory and put it in separate module so that can be used when needed. It adds a lot of additional dependencies when used.

Will need some time to update code, raylib-go will have just cmd util, you are working on real tool there :)

[gen2brain]

Any plans maybe for function/s that should load Music files from resource? I think I will have some problems there in Go.

[raysan5]

Hi @gen2brain! Thanks for your quick answer!

Looks great, I would prefer to go without that crc32 if possible, only complicates things it seems.

I added that field after lot of reading on fileformats, ZIP and PNG use it for data corruption detection, PNG specs also provide C code about implementation. CRC is also recommended here and here or even here.

The truth is that there is not much information out there on designing custom file-formats, it seems very case-specific and information usually points to 3 well-designed formats: RIFF, PNG and ZIP.

So rres.h is not required anywhere anymore, right?

That's right. Using it is a user decision.

...raylib-go will have just cmd util, you are working on real tool there

Just trying to create a simple raygui-based tool for rres generation (also usable from command line), similar to MonoGame Content Pipeline tool.

Any plans maybe for function/s that should load Music files from resource?

Still thinking on that issue... even XNB file-format uses external WMA access, referenced from XNB file. Embedding OGG music files into a .rres won't be a good idea...

[gen2brain]

Thanks, that was a good read. I am no expert here, I just passed what I thought I see at that moment, but now I can understand how and where CRC can help.

I actually work as sysadmin, programming is just a hobby, but I must say that since I started playing with raylib I feel like I upgraded to new level and got some new skills :) And I only learned some C through Python ctypes and Go cgo, nothing serious.

Music files can stay where they are and don't need to be bundled if that is simpler, and it looks like it is a much simpler to do that. Usually user will not have many of those and it is not big deal if they are not bundled in resources.

Btw. I recently played with ZIP format, I wanted to corrupt archives on purpose, but in different ways, so I can use them in tests, I found out that https://github.com/evanmiller/hecate is a great tool for that, much better that any other hex editor I found.

We can continue discussion on new rres github project in the future.

[raysan5]

Moving this issue to rres project. Check raysan5/rres#1, raysan5/rres#2, raysan5/rres#3.