/AppImageKit

Native distro packages for appimaged #339

Open
probonopd opened this Issue Jan 28, 2017 · 25 comments

Comments

Assignees
No one assigned
Labels
enhancement help-wanted
Projects
None yet
Milestone
No milestone
4 participants
@probonopd @marguerite @rickysarraf @TheAssassin
@probonopd
Owner

probonopd commented Jan 28, 2017 •
Edited 1 time

EDIT: Anyone reading this and interested in doing native distribution packages, we are on IRC on #AppImages on irc.freenode.net

@rickysarraf writes in probonopd#325

I can speak on what is the case in Debian. I'm not sure about other distributions. ON Debian, on the buildd infrastructure, network access is disabled. And there are good reasons to do that. Your git submodules are just defined as a git repository.

I am not sure what is the best path of action here, but I will definitely support anyone who wants to make a native (in the sense of: part of the official debian distribution) debian package for appimaged. Also see the great work by @marguerite who has packaged it (and appimagetool) for openSUSE.

Perhaps @ScarlettGatelyClark wants to chime in?

Collecting here the need for a debian package:

@probonopd probonopd referenced this issue Jan 28, 2017

Open

Build scripts are fragile (wrong arch in systemd-nspawn) #325

@probonopd probonopd added enhancement help-wanted labels Jan 28, 2017

@marguerite

marguerite commented Jan 28, 2017

In openSUSE Build Service, we don't have network access in the build vitual machine created dynamically either. But we have something called source service that can git with sub modules and compress the result to a tarbal. So running git submodule directly when building is also impossible here. It has to be commented out for native packaging.

But in bash there are ways to detect network availability. So I think we could use git only when network is possible. on those isolated machines, just unpack the tarball. Leaving the responsibility to create the tarball to the distro packagers
@marguerite

marguerite commented Jan 28, 2017

I think checking the common local IP could be enough. ping or nc is too heavy in this case.
@rickysarraf

rickysarraf commented Jan 29, 2017

There are 2 parts of the problem here.

  1. Network access. This can be easily solved. Either upstream does point releases, with tarballs, or else I can myself take git snapshots and create a tarball.

  2. forked sub-modules: There's a forked squashfs-tools git submodule. Under debian squashfs-tools is already packaged. And I'm not going to carry this forked module inside appimagekit. You could either push your change to the squashfs folks and add a versioned build dependency here. Or you could just carry the delta and ask the user to fetch the squashfs-tools source themselves. I'd suggest you push your changes, because otherwise even if you carry the delta, it becomes difficult for me to push into a pristine source package in Debian.

So to summarize point 2, the concern is not about a sub-module, but more about carrying embedded copies of the source.

The current build script structure is trying too hard being simple for the end user. In doing that, it is breaking standard practices. In my opinion, you should keep them separately as end-user-helper-scripts and instead have a much standardized build setup.

I doubt if end-users are going to build the tool. Most they care is is a binary package.
@probonopd
Owner

probonopd commented Jan 29, 2017 •
Edited 1 time

@rickysarraf

There's a forked squashfs-tools git submodule. Under debian squashfs-tools is already packaged. And I'm not going to carry this forked module inside appimagekit

Yes, I am not happy about that.

You could either push your change to the squashfs folks

Sent a PR half a year ago but no reaction: plougher/squashfs-tools#13 - is something wrong with it?

Or you could just carry the delta and ask the user to fetch the squashfs-tools source themselves

Do you mean "carry the delta" as in "provide a diff"?

Or do you think we should send a patch to https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=squashfs-tools?
@rickysarraf

rickysarraf commented Jan 29, 2017

@probonopd
Owner

probonopd commented Jan 29, 2017

Thanks @rickysarraf
@rickysarraf

rickysarraf commented Jan 29, 2017

You're welcome. Let's hope it gets merged.

Once that is done, I can care for its (and squashfuse) inclusion into Debian.
@rickysarraf

rickysarraf commented Jan 31, 2017

@rickysarraf

rickysarraf commented Jan 31, 2017

@rickysarraf

rickysarraf commented Jan 31, 2017

@probonopd
Owner

probonopd commented Feb 4, 2017

Is there any action required from my side at this time?
@probonopd
Owner

probonopd commented Feb 9, 2017

Yay, it just was merged: plougher/squashfs-tools#13 (comment)
@rickysarraf

rickysarraf commented Feb 10, 2017

Great news. I'll soon look at the debian/ packaging again.
@rickysarraf

rickysarraf commented Feb 11, 2017

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854901
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854902

For anyone interested to help/collaborate, please feel free to drop me a note here, or on the referenced Debian ITPs.
@probonopd
Owner

probonopd commented Feb 11, 2017 •
Edited 1 time

Thank you @rickysarraf.

The primary objective should be to get a native package for appimaged which is one part of AppImageKit.

Minor disagreement, I don't think "Flatpak and Snaps solve similar problems". They don't allow users to download a single file from an application's download page, and run that on most Linux distributions without changes to the system. I think they are valuable, but they solve different problems.
@probonopd
Owner

probonopd commented Feb 11, 2017

ping @ScarlettGatelyClark on http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854902
@rickysarraf

rickysarraf commented Feb 11, 2017

@probonopd probonopd changed the title from Native debian package for appimaged to Native distro packages for appimaged Feb 11, 2017

@probonopd
Owner

probonopd commented Feb 11, 2017 •
Edited 1 time

Also see https://build.opensuse.org/package/show?project=devel%3Atools%3Abuilding&package=appimagekit by @dirkmueller (thanks @kossebau for the hint)
@rickysarraf

rickysarraf commented Jul 31, 2017

@probonopd I went off for quite some time. Real life priorities came in.

I have packaged the squashfuse dependency as a package now. But now, with version 9, I see the entire Cmake based build to have vanished. Is build.sh the way upstream way to build appimagekit ?
@TheAssassin
Member

TheAssassin commented Jul 31, 2017

Yes, build.sh is the most upstream way to create binaries. Though, it is pretty likely that you will need only parts of it, so you should think about copy-pasting to your own script for building only the parts you need.

I am working to create and improve CMake configs for the AppImage ecosystem (primarily for tool development and not necessarily for release buils), suggestions welcome.
@rickysarraf

rickysarraf commented Jul 31, 2017

@TheAssassin You guys should have revived the build.sh script such that the actual compilation and 3rd party library fetches were separate.

@probonopd You still applying squashfuse patches. Weren't these included upstream the last time we visited this topic ?

cd squashfuse
if [ ! -e ./ll.c.orig ]; then
  patch -p1 --backup < ../squashfuse.patch
  patch -p1 --backup < ../squashfuse_dlopen.patch
fi
if [ ! -e ./squashfuse_dlopen.c ]; then
  cp ../squashfuse_dlopen.c .
fi
if [ ! -e ./squashfuse_dlopen.h ]; then
  cp ../squashfuse_dlopen.h .
fi

@probonopd Also, that opensuse build link you've shared, is broken.
@rickysarraf

rickysarraf commented Jul 31, 2017

So this is the complete build rules for appimagekit, excluding the dependencies. If I am to package it for Debian, I'd just adapt these. But it'd be nicer if this was done upstream.

# Compile runtime but do not link

$CC -DVERSION_NUMBER=\"$(git describe --tags --always --abbrev=7)\" -I../squashfuse/ -D_FILE_OFFSET_BITS=64 -g -Os -c ../runtime.c

# Prepare 1024 bytes of space for updateinformation
printf '\0%.0s' {0..1023} > 1024_blank_bytes

objcopy --add-section .upd_info=1024_blank_bytes \
        --set-section-flags .upd_info=noload,readonly runtime.o runtime2.o

objcopy --add-section .sha256_sig=1024_blank_bytes \
        --set-section-flags .sha256_sig=noload,readonly runtime2.o runtime3.o

# Now statically link against libsquashfuse_ll, libsquashfuse and liblzma
# and embed .upd_info and .sha256_sig sections
$CC -o runtime ../elf.c ../notify.c ../getsection.c runtime3.o \
    ../squashfuse/.libs/libsquashfuse_ll.a ../squashfuse/.libs/libsquashfuse.a ../squashfuse/.libs/libfuseprivate.a \
    -L../xz-5.2.3/build/lib -Wl,-Bdynamic -lpthread -lz -Wl,-Bstatic -llzma -Wl,-Bdynamic -ldl
$STRIP runtime

# Test if we can read it back
readelf -x .upd_info runtime # hexdump
readelf -p .upd_info runtime || true # string

# The raw updateinformation data can be read out manually like this:
HEXOFFSET=$(objdump -h runtime | grep .upd_info | awk '{print $6}')
HEXLENGTH=$(objdump -h runtime | grep .upd_info | awk '{print $3}')
dd bs=1 if=runtime skip=$(($(echo 0x$HEXOFFSET)+0)) count=$(($(echo 0x$HEXLENGTH)+0)) | xxd

# Insert AppImage magic bytes

printf '\x41\x49\x02' | dd of=runtime bs=1 seek=8 count=3 conv=notrunc

# Convert runtime into a data object that can be embedded into appimagetool

ld -r -b binary -o data.o runtime

# Test if we can read it back
readelf -x .upd_info runtime # hexdump
readelf -p .upd_info runtime || true # string

# The raw updateinformation data can be read out manually like this:
HEXOFFSET=$(objdump -h runtime | grep .upd_info | awk '{print $6}')
HEXLENGTH=$(objdump -h runtime | grep .upd_info | awk '{print $3}')
dd bs=1 if=runtime skip=$(($(echo 0x$HEXOFFSET)+0)) count=$(($(echo 0x$HEXLENGTH)+0)) | xxd

# Convert runtime into a data object that can be embedded into appimagetool

ld -r -b binary -o data.o runtime

# Compile appimagetool but do not link - glib version

$CC -DVERSION_NUMBER=\"$(git describe --tags --always --abbrev=7)\" -D_FILE_OFFSET_BITS=64 -I../squashfuse/ \
    $(pkg-config --cflags glib-2.0) -g -Os ../getsection.c  -c ../appimagetool.c

# Now statically link against libsquashfuse - glib version
if [ $STATIC_BUILD -eq 1 ]; then
  # statically link against liblzma
  $CC -o appimagetool data.o appimagetool.o ../elf.c ../getsection.c -DENABLE_BINRELOC ../binreloc.c \
    ../squashfuse/.libs/libsquashfuse.a ../squashfuse/.libs/libfuseprivate.a \
    -L../xz-5.2.3/build/lib \
    -Wl,-Bdynamic -ldl -lpthread \
    -Wl,--as-needed $(pkg-config --cflags --libs glib-2.0) -lz -Wl,-Bstatic -llzma -Wl,-Bdynamic
else
  # dinamically link against distro provided liblzma
  $CC -o appimagetool data.o appimagetool.o ../elf.c ../getsection.c -DENABLE_BINRELOC ../binreloc.c \
    ../squashfuse/.libs/libsquashfuse.a ../squashfuse/.libs/libfuseprivate.a \
    -Wl,-Bdynamic -ldl -lpthread \
    -Wl,--as-needed $(pkg-config --cflags --libs glib-2.0) -lz -llzma
fi

# Version without glib
# cc -D_FILE_OFFSET_BITS=64 -I ../squashfuse -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -g -Os -c ../appimagetoolnoglib.c
# cc data.o appimagetoolnoglib.o -DENABLE_BINRELOC ../binreloc.c ../squashfuse/.libs/libsquashfuse.a ../squashfuse/.libs/libfuseprivate.a -Wl,-Bdynamic -ldl -lpthread -lz -Wl,-Bstatic -llzma -Wl,-Bdynamic -o appimagetoolnoglib

# Compile and link digest tool

if [ $STATIC_BUILD -eq 1 ]; then
  $CC -o digest ../getsection.c ../digest.c -I../openssl-1.1.0c/build/include -L../openssl-1.1.0c/build/lib \
    -Wl,-Bstatic -lssl -lcrypto -Wl,-Bdynamic -lz -ldl
else
  $CC -o digest ../getsection.c ../digest.c -Wl,-Bdynamic -lssl -lcrypto -lz -ldl
fi

$STRIP digest

# Compile and link validate tool

if [ $STATIC_BUILD -eq 1 ]; then
  $CC -o validate ../getsection.c ../validate.c -I../openssl-1.1.0c/build/include -L../openssl-1.1.0c/build/lib \
    -Wl,-Bstatic -lssl -lcrypto -Wl,-Bdynamic -Wl,--as-needed $(pkg-config --cflags --libs glib-2.0) -lz -ldl
else
  $CC -o validate ../getsection.c ../validate.c -Wl,-Bdynamic -lssl -lcrypto \
    -Wl,--as-needed $(pkg-config --cflags --libs glib-2.0) -lz -ldl
fi

$STRIP validate

# AppRun
$CC ../AppRun.c -o AppRun

# check for libarchive name
have_libarchive3=0
archive_n=
if printf "#include <archive3.h>\nint main(){return 0;}" | cc -w -O0 -xc - -Wl,--no-as-needed -larchive3 2>/dev/null ; then
  have_libarchive3=1
  archive_n=3
fi
rm -f a.out

# appimaged, an optional component
if [ $STATIC_BUILD -eq 1 ]; then
  $CC -std=gnu99 -o appimaged -I../squashfuse/ ../getsection.c ../notify.c ../elf.c ../appimaged.c \
    -D_FILE_OFFSET_BITS=64 -DHAVE_LIBARCHIVE3=$have_libarchive3 -DVERSION_NUMBER=\"$(git describe --tags --always --abbrev=7)\" \
    ../squashfuse/.libs/libsquashfuse.a ../squashfuse/.libs/libfuseprivate.a \
    -L../xz-5.2.3/build/lib -I../inotify-tools-3.14/build/include -L../inotify-tools-3.14/build/lib \
    -Wl,-Bstatic -linotifytools -Wl,-Bdynamic -larchive${archive_n} \
    -Wl,--as-needed \
    $(pkg-config --cflags --libs glib-2.0) \
    $(pkg-config --cflags --libs gio-2.0) \
    $(pkg-config --cflags --libs cairo) \
    -ldl -lpthread -lz -Wl,-Bstatic -llzma -Wl,-Bdynamic
else
  $CC -std=gnu99 -o appimaged -I../squashfuse/ ../getsection.c ../notify.c ../elf.c ../appimaged.c \
    -D_FILE_OFFSET_BITS=64 -DHAVE_LIBARCHIVE3=$have_libarchive3 -DVERSION_NUMBER=\"$(git describe --tags --always --abbrev=7)\" \
    ../squashfuse/.libs/libsquashfuse.a ../squashfuse/.libs/libfuseprivate.a \
    -Wl,-Bdynamic -linotifytools -larchive${archive_n} \
    -Wl,--as-needed \
    $(pkg-config --cflags --libs glib-2.0) \
    $(pkg-config --cflags --libs gio-2.0) \
    $(pkg-config --cflags --libs cairo) \
    -ldl -lpthread -lz -llzma
fi

cd ..

# Strip and check size and dependencies

rm build/*.o build/1024_blank_bytes
$STRIP build/* 2>/dev/null
chmod a+x build/*
ls -lh build/*
for FILE in $(ls build/*) ; do
  echo "$FILE"
  ldd "$FILE" || true
done

bash -ex "$HERE/build-appdirs.sh"

ls -lh

mkdir -p out
cp -r build/* ./*.AppDir out/
@probonopd
Owner

probonopd commented Jul 31, 2017 •
Edited 1 time

@@TheAssassin You guys should have revived the build.sh script such that the actual compilation and 3rd party library fetches were separate.

Wouldn't it be sufficient to (pseudo-code)

if not (building_debian_package){
  do stuff that you don't need
}

I'd happily merge a PR with such if statements as long as it doesn't alter my builds and only gets in effect if your debian source package is used.

@probonopd You still applying squashfuse patches. Weren't these included upstream the last time we visited this topic ?

Were all of them? I was only aware of the offset patch having been accepted.
In any case, what we have now is working for me. PR welcome if you think it needs to be changed.

@probonopd Also, that opensuse build link you've shared, is broken.

https://build.opensuse.org/package/show/devel:tools:building/appimagekit seems to be the official one now.
@probonopd
Owner

probonopd commented Jul 31, 2017 •
Edited 1 time

So this is the complete build rules for appimagekit

Isn't this the build.sh script with some lines removed? If so, please use if statements as described above.
@TheAssassin
Member

TheAssassin commented Jul 31, 2017 •
Edited 1 time

As you might have noticed, there's also a CMake configuration that we could extend and modify to work for you, @rickysarraf. If you just need appimaged, let me add the configuration and we're good to go.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment