Develop libdwarfs (like libsquashfs) that will allow to use dwarfs without fuse

[maxirmx]

No description provided.

[maxirmx]

Issues to be resolved in order to apply libdwarfs for tebako ruby packager

1. dwarfs is C++ library. Ruby is C program. Normally calling C++ from a program with C start-up is not possible. So some kind of C++ startup wrapper will be required in order to link libdwarfs to ruby.
2. dwarfs code is not ported to Windows and Mac. Porting may be a challenge because of multiple dependencies on complex Facebook and Google
3. dwarfs implements inode operations. Ruby requires file operations. Practically it means that that libdwarfs needs to implement fuse functionality -- translate file operations to inode metadata and inode operations
4. inodes is not something supported by Windows. It is not clear whether it is an issue

[ronaldtse]

@maxirmx Will a structure like this work?

Typical FUSE

Tebako with internal and external image redirection

[maxirmx]

It will work.
However as far as I understood glibc redirection (as in retrace) requires dynamic linking (shared library)
You will have to distribute a package that includes several files

[maxirmx]

What we have as of Sept 25th 2021:

We have it because the project started with tamatebako/tebako#2

[maxirmx]

This will be "organic" change from using libdwarfs with FUSE wrapper towards using libdwarfs with out new wrapper

[maxirmx]

@ronaldtse we already have libdwarfs as explained above. This new libray will be libdwarfs wrapper that will provide inode <-> fd description. I believe it requires some other name. In ruby packer they just copy-paste files from squashFS but I do no believe it makes any sence.

Also the idea with glibc redirector contradicts your requirement to have statically lined single binary. Both requirement can be met but not together.

[ronaldtse]

the idea with glibc redirector contradicts your requirement to have statically lined single binary. Both requirement can be met but not together.

I understand the potential contradiction. The goal is to have a single binary wrapping that allows a Ruby program with its dependencies to run, so it's not really about static / dynamic. If a dynamically compiled Ruby is linked with other dependencies wrapped within a Tebako package, to the user they are self-contained and hence achieving what we need.

[ronaldtse]

What we have as of Sept 25th 2021:
We have it because the project started with tamatebako/tebako#2

My only concern is that it requires the FUSE kernel module to be loaded, by default Linux and Macs don't have that enabled. So it's just the next step to also move that portion into user-space.

[maxirmx]

I understand the potential contradiction. The goal is to have a single binary wrapping that allows a Ruby program with its dependencies to run, so it's not really about static / dynamic. If a dynamically compiled Ruby is linked with other dependencies wrapped within a Tebako package, to the user they are self-contained and hence achieving what we need.

To access dwarFS image ruby needs glibc shared library that you want to place inside dwarFS image. This does not seem possible

[maxirmx]

My only concern is that it requires the FUSE kernel module to be loaded, by default Linux and Macs don't have that enabled. So it's just the next step to also move that portion into user-space.

It is possible to eliminate FUSE dependency. I can patch ruby in the same way ruby-packer does.
(Actually I believe I know how to achieve much better packager performance)

[ronaldtse]

Patching the Ruby interpreter is acceptable.

Right now we already have a reasonable way of getting things working by:

1. Patching the Ruby interpreter.
2. The patched Ruby interpreter can be pre-built (once, by us) for all users

I was wondering whether we can "insert" a layer between the Ruby interpreter and its file reading functions.

[maxirmx]

I was wondering whether we can "insert" a layer between the Ruby interpreter and its file reading functions.

We will. It is called "ANewLibrary" in my diagram.
The difference we are discussing is the following:

• with glibc module we would be patching glibc.so (binary, dynamically linked)
• with ANewLibrary we will patch ruby interpreter source so that it will call "ANewLibrary" (statically linked)

[ronaldtse]

with glibc module we would be patching glibc.so (binary, dynamically linked)
with ANewLibrary we will patch ruby interpreter source so that it will call "ANewLibrary" (statically linked)

What are the considerations for better / more extensible for the future?

[maxirmx]

What are the considerations for better / more extensible for the future?

No opinion yet. Both involve some unconventional patching.
Anyway I have to start with a library that wraps existing libdwarfs and provides different interface.
Ruby or glibc patching is the next step. The wrapper is the same.

[maxirmx]

The list of C functions that shall be supported for Ruby packaging: #3