Desktop Integration Data without accessing payload

[lawl]

Stolen from here

type 3 should specify a way to extract desktop integration data without having to read the payload format as a fallback.

Absolutely. Rationale:

• FileManagers may want to generate thumbnails. Requiring a squashfs driver to pull a tiny PNG out of a file seems unreasonable.
• Other software may also have good reasons to access metadata such as update URLs, icons and .desktop files to be able to easily integrate AppImages into the desktop and keep them up to date.
• Should the imaging format from ever change a way from squashfs, it may be easier to do so in a backward compatible way as most external tools will never need to care about the payload content.

[TheAssassin]

Requiring a squashfs driver to pull a tiny PNG out of a file seems unreasonable.

Please stop claiming that desktop integration was just about a single "tiny" file. That's the case only with thumbnailing, which is only a small part of the desktop integration. We are talking about directory trees with many pictures, one or more desktop files, MIME definitions, ...

update URLs

Those were never contained inside the payload.

I also don't think it's a big problem to have to actually extract data from the payload. libappimage demonstrates how this can be done, and has been working fine for years already. For instance, AppImageLauncher does not need to know anything about payload formats, since it just asks libappimage to do the job.

Also, desktop integration itself is not just about extracting files. AppImageLauncher for instance implements some custom additional logic that has never made its way into libappimage. For instance, it runs several cache update commands (update-mime-database, gtk-update-icon-cache, ...) to make changes visible instantly.

[lawl]

Please stop claiming that desktop integration was just about a single "tiny" file

I never claimed that, you interpreted that.

We are talking about directory trees with many pictures

Other operating systems have solved that decades ago and it doesn't require a sqashfs driver, see .rsrc.

libappimage demonstrates how this can be done, and has been working fine for years already. For instance, AppImageLauncher does not need to know anything about payload formats, since it just asks libappimage to do the job.

Hiding the smell in a lib does not make it less ridiculous.

[mgord9518]

Actually, in the other discussion about .zips, I got an idea. What if the AppImage header was just a small zip file, containing desktop integration files and the update info? Being that zip doesn't care where its magic numbers are, it could essentially be placed at any arbitrary point of the AppImage, allowing for very flexible construction of the files.

This would make it incredibly easy to extract the information, as desktop integration/updating software would simply open the AppImage as a zip file. This wouldn't require a SquashFS driver, it wouldn't require any ELF hacking, and could simplify desktop integration software by foregoing the need to use blobs to query files as all the files inside the archive could have established names. Eg: 'desktopEntry' instead of 'firefox.desktop'.

[TheAssassin]

Much better idea than using zip for the payload. Do you have any docs on "zip doesn't care where its magic numbers are"? I'm curious, I couldn't find anything about that before.

[mgord9518]

For some reason it's a bit hard to find, but in the wikipedia article under the "Design" section, there is a small paragraph mentioning it called "Combination with other file formats" which mentions it. You can also find it by searching "zip inside image steganography", which is where I originally learned about it years ago.

I'd like to confirm it's actually part of the PKWARE standard and not just being common practice in implementations though.

Edit: here's another mention in the Wikipedia article (also under the "Design" section):
"There is no BOF or EOF marker in the ZIP specification. Conventionally the first thing in a ZIP file is a ZIP entry, which can be identified easily by its local file header signature. However, this is not necessarily the case, as this not required by the ZIP specification - most notably, a self-extracting archive will begin with an executable file header."

[TheAssassin]

Ah, that's an interesting information. I guess I never tried manually extracting an SFX then. I remember those from Windows...

Well, perhaps you could put together a little test image (I guess cat runtime-x86_64 some.zip > test.img will do) and try it with various implementations? The most relevant ones are InfoZip (zip tool), 7-zip (I'm confident it will work), libzip, Python's zipfile and libarchive, those should be the most widely used libraries.

[mgord9518]

Went over the top just to see how extreme it allows for by simply appending a zip to the FreeCAD AppImage, which is almost 1GB in size.

InfoZip, Python stdlib implementation and BSD tar all work perfectly

Unfortunately, the Go stdlib doesn't work in this way, nor does libzip, but it should be pretty easy to scan the file for the header for situations like this

[mgord9518]

Alright, so here's a really interesting update. I've done a bit of research into this and apparently, InfoZip has a flag called -A, which as the manpage explains, "adjusts the suffix", inserting some extra info that tells zip implementations that this is supposed to be a self-extracting archive.

Well, this magical information seems to be vital for the aforementioned "non-working" implementations, as simply running zip -A appWithZipAppended.AppImage fixes them with zero workarounds needed.

I've since retested with Go, Python, ARK (libzip), InfoZip, BSD tar (libarchive) and all work as if the AppImage is just a normal zip file, regardless of size. The only exception is p7zip, and not because it has some issue reading the appended zip archive, but simply because it sees the SquashFS header first, assuming that it's intended to be read as SquashFS. The file doesn't even get scanned through as they just check the footer, so extracting is as fast as it would be on a plain zip file.

It's also important to also note that the zip archive MUST be applied to the end of the AppImage, not between the runtime and SquashFS image, but this shouldn't change anything.

Here's some other cool things: AppImages built this way are still readable by type 2 integration software assuming they're ELF and use gzip compression. Adopting mkappimage, appimagetool, and libappimage to work with them be trivial as they'd only need a zip library, and newly built AppImages could use whatever language, compression etc. without needing to worry about being incompatible with desktop integration software or AppImageUpdate.

[mgord9518]

@TheAssassin Any new thoughts on this discussion?