Linux and macOS releases are missing -- static linking version preferred

[kristofmulier]

I've very happy to find the latest releases here:

https://github.com/avrdudes/avrdude/releases

Thanks a lot @mariusgreuel and @dl8dtl for your awesome work! Unfortunately, I can't see any Linux releases. The Linux releases are available in the nightly builds though:

https://github.com/avrdudes/avrdude/wiki/Getting-Nightly-Builds-for-AVRDUDE

It would be great if they would also be available on the release page. I know that usually the Linux distros package their own binaries to fit into their package systems (RPM, dpkg, ...). But some people prefer to get the builds directly from GitHub. It would also be helpful for our usecase: we'd like to download AVRDUDE directly to the computer of our users (the users of Embeetle IDE, a new microcontroller IDE) without going through a package manager.

Thanks a lot ^_^

[mcuee]

The current github action build of avrdude may not be suitable as a Linux release, lower/older version of Linux [eg: Ubuntu 18.04 or Debian 8) may be better. And different Linux distro may have different dependancies. So I am not sure how practical this is.
Ref: https://github.com/avrdudes/avrdude/blob/main/.github/workflows/build.yml (Ubuntu:latest and Debian 11).

For example, we only provide Windows binary for libusb projects.

[ygramoel]

I just managed to build avrdude on a virtual machine running Debian Jessie.

It did require a recent version of cmake (3.21.4) that I had to install from source as well.
Except for that, the standard instructions on github worked:

$ sudo apt install flex bison ...
$ git clone https://github.com/avrdudes/avrdude
$ cd avrdude
$ cmake -D CMAKE_BUILD_TYPE=RelWithDebInfo -B build_linux
$ cmake --build build_linux

For distribution, I checked the dependencies using ldd:

$ ldd build_linux/src/avrdude
	linux-vdso.so.1 (0x00007ffdc0de5000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff785365000)
	libusb-0.1.so.4 => /lib/x86_64-linux-gnu/libusb-0.1.so.4 (0x00007ff78515c000)
	libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007ff784f44000)
	libhidapi-libusb.so.0 => /usr/lib/x86_64-linux-gnu/libhidapi-libusb.so.0 (0x00007ff784d3d000)
	libftdi.so.1 => /usr/lib/x86_64-linux-gnu/libftdi.so.1 (0x00007ff784b36000)
	libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007ff7848ec000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff784541000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007ff784332000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff784115000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff783f0d000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007ff783ce3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff785666000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff783adf000)

For maximum compatibility with different Linux distros, I am tempted to link these libraries statically:

/usr/lib/x86_64-linux-gnu/libusb.a     
/usr/lib/x86_64-linux-gnu/libusb-1.0.a
/usr/lib/x86_64-linux-gnu/libhidapi-libusb.a
/usr/lib/x86_64-linux-gnu/libftdi.a

Would you suggest any other libraries for static linking? For example, libudev or libreadline?

[dl8dtl]

OTOH, if you have an old reference system already available, could you just give the nightly builds a try?

If they work even on your relatively old system, I'd just pick those. I don't think the APIs of the libraries you mentioned have been changed much lately.

The bigger issue is probably that the "modern" build might require a certain LIBC_foo version symbol, for whatever reason.

[mcuee]

@ygramoel
I think it is enough to statically link to the following.

/usr/lib/x86_64-linux-gnu/libusb.a     
/usr/lib/x86_64-linux-gnu/libusb-1.0.a
/usr/lib/x86_64-linux-gnu/libhidapi-libusb.a
/usr/lib/x86_64-linux-gnu/libftdi.a

[ygramoel]

Linking the four libraries mentioned above statically works for me. The others, including libudev, are linked dynamically:

$ ldd avrdude
	linux-vdso.so.1 (0x00007ffc1d244000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fddcebd2000)
	libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007fddce988000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fddce76b000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fddce55c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fddce1b1000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fddcdf87000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fddceed3000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fddcdd7f000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fddcdb7b000)

This is the link command I used:

/usr/bin/cc -O2 -g -DNDEBUG -L/home/johan/local/openssl/lib -L/lib/x86_64-linux-gnu -Wl,-rpath,/home/johan/local/openssl/lib CMakeFiles/avrdude.dir/main.c.o CMakeFiles/avrdude.dir/term.c.o CMakeFiles/avrdude.dir/whereami.c.o -o avrdude  libavrdude.a -lm /usr/lib/x86_64-linux-gnu/libusb.a /usr/lib/x86_64-linux-gnu/libusb-1.0.a /usr/lib/x86_64-linux-gnu/libhidapi-libusb.a /usr/lib/x86_64-linux-gnu/libftdi.a /usr/lib/x86_64-linux-gnu/libreadline.so -lpthread -ludev

I had to sudo apt install libudev-dev first; your mileage may vary.

It is most probably not a good idea to link libdl or libc statically. I am less certain about the others.

[dl8dtl]

I just tried it on an elderly Ubuntu 16.04LTS:

7% ldd avrdude
./avrdude: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./avrdude)
        linux-vdso.so.1 =>  (0x00007ffecb7c1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f895a1f7000)
        libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f8959fdf000)
        libusb-0.1.so.4 => /lib/x86_64-linux-gnu/libusb-0.1.so.4 (0x00007f8959dd6000)
        libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f8959bbe000)
        libhidapi-libusb.so.0 => /usr/lib/x86_64-linux-gnu/libhidapi-libusb.so.0 (0x00007f89599b7000)
        libftdi1.so.2 => /usr/lib/x86_64-linux-gnu/libftdi1.so.2 (0x00007f89597a9000)
        libreadline.so.8 => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f89593df000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f895a500000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f89591c5000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f895a63c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8958fa8000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8958da0000)

As you can see, it's not the libusb or libftdi stuff that is causing any trouble. It's that "minimum version" symbol in libc, and libreadline.

[kristofmulier]

I just tried the nightly build on this old Debian Virtual Machine - Debian GNU/Linux 8.11 (jessie). Apparently, two libraries are missing:

johan@debian:~/avrdude_nightly$ ./avrdude
./avrdude: error while loading shared libraries: libftdi1.so.2: cannot open shared object file: No such file or directory

johan@debian:~/avrdude_nightly$ ldd avrdude
./avrdude: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./avrdude)
	linux-vdso.so.1 (0x00007ffdfcef8000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbe8139f000)
	libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007fbe81189000)
	libusb-0.1.so.4 => /lib/x86_64-linux-gnu/libusb-0.1.so.4 (0x00007fbe80f80000)
	libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007fbe80d68000)
	libhidapi-libusb.so.0 => /usr/lib/x86_64-linux-gnu/libhidapi-libusb.so.0 (0x00007fbe80b61000)
	libftdi1.so.2 => not found
	libreadline.so.8 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbe807b6000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbe816a0000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fbe805a7000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbe8038a000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fbe80182000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbe7ff7e000)

[dl8dtl]

OK, so your system's libftdi1 is even older than Ubuntu 16, it seems. ;-)

The biggest issue is libc. Everything else, you can just ship the .so that has been used to build along with the binary, and set LD_LIBRARY_PATH. (If you ship your binary without the .so files, it e.g. won't find an appropriate libreadline on a more modern system.)

[dl8dtl]

Try adding librreadline to your above list of static libraries, and see whether the result will run on a current Linux distribution.

[ygramoel]

Linking readline statically works; I had to add -ltinfo to the link command.

/usr/bin/cc -O2 -g -DNDEBUG -L/home/johan/local/openssl/lib -L/lib/x86_64-linux-gnu -Wl,-rpath,/home/johan/local/openssl/lib CMakeFiles/avrdude.dir/main.c.o CMakeFiles/avrdude.dir/term.c.o CMakeFiles/avrdude.dir/whereami.c.o -o avrdude  libavrdude.a -lm /usr/lib/x86_64-linux-gnu/libusb.a /usr/lib/x86_64-linux-gnu/libusb-1.0.a /usr/lib/x86_64-linux-gnu/libhidapi-libusb.a /usr/lib/x86_64-linux-gnu/libftdi.a /usr/lib/x86_64-linux-gnu/libreadline.a -ltinfo -lpthread -ludev

It runs on my Linux Mint 19 Tara, but not on a more recent Ubuntu 21.10. The problem is libtinfo:

$ ./avrdude 
./avrdude: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory

[ygramoel]

The original version, with both libreadline and libtinfo linked dynamically, doesn't work on a recent Ubuntu either. However, linking both libreadline and libtinfo statically works.

[kristofmulier]

I'm Johan's (@ygramoel) colleague. Here is the binary he just made:

https://new.embeetle.com/downloads/beetle_tools/linux/avrdude_7.0.0_feaa1c6_64b.7z

It's a 7z compressed folder containing two files:

• avrdude (the executable)
• avrdude.conf

Can you please give it a try on your Linux machine(s)?

[dl8dtl]

Works here, customized Ubuntu 20.04.
Can test that old Ubuntu 16.04 tonight.

[mariusgreuel]

@kristofmulier Just to add some history and reasoning to this discussion, even if it may be redundant: The current CMake setup and Github Actions do not support building Linux binaries that are suitable for download, which is why they do not show up in the release artifacts. You can tell from the discussion that your requirement is not just a matter of including the build artifacts in the release, but actually quite a bit of work to get it right.

Technically speaking, the whole build pipeline consists of three things: a) the CMake build system b) the GitHub build actions c) the GitHub release actions

Regarding a) The current CMake setup is designed to support the two most common scenarios: 1) Building Un*x binaries on a defined target system using dynamic linking, suitable for package managers etc., and 2) Building Windows binaries suitable for download using static linking, supporting Windows 7 and up.

See also: #960 (comment)

Regarding b) The GitHub build action for Linux is merely a build check, with the intention that the build works on a current Linux system and toolset, throwing no errors and warnings, etc, for the scenario outlined in a). With that reasoning in mind, I picked the latest Debian distro (i.e. the VM 'ubuntu-latest') with the latest GCC toolset and the latest libraries.

Regarding c) As the Linux binaries are not meant for downloading, only the Windows release artifacts (the MSVC builds with the custom libraries) are included in the release.

Now, we have a few new requirements here, such as the Arduino team wishing for statically linked binaries that they can ship with the Arduino IDE (#946 (reply in thread)), which is probably what you expect from a Linux binary offered on the release page, or others finding value in building dynamically linked binaries for Windows (#954). All these requests are related, because they all require changes in a), b), and c).

You are welcome to submit a PR that addresses your needs, however, I would like to point out that the requirements for building Linux binaries suitable for download conflict with the requirements that I outlined above (because you want static linking, support for older distros, etc.). So when you submit a PR, please keep the existing build and CI actions intact, and add a extra jobs that builds the required outputs.

[dl8dtl]

Works on Ubuntu 16.04 as well.

[mcuee]

Personally I am not in favor of providing the "official linux binary" in this repo as it does increase the Maintenance Efforts and I do not see much values to providing binaries to Linux and macOS users as there are other more official and popular channels to do that (eg: Linux distros, Arduino, macports and homebrew). But I will leave the decision to the admins.

On the other hand, I think it is good to test static link version for Linux and macOS in the github action. Right now MinGW and MSVC use static linking (we should keep that) and Linux and macOS use dynamic linking (we should keep that as well but add static linking testing as well).

[mcuee]

For those who want to build their own Linux static linking binary, Arduino cross-build container and the Arduino-packing project build script are a good one to use.
Ref: https://github.com/arduino/avrdude-packing/blob/main/.github/workflows/release.yml

I have no issues to build 32bit Linux and macOS binary with avrdude git main using the container. You can see you do not need to use very complicated script. And only one patch is needed for Linux and macOS.

• Enhance CMake to be able to support static link for Linux and macOS #1159 (comment)

I have some issues under mingw32 but hopefully @umbynos can help out (he is able to build under mingw32).

@kristofmulier @MCUdude and @SpenceKonde,
I think the above container is a good one to use. You can even build custom avrdude version, with your own branding or patches if you prefer. Then you do not need to rely on Arduino project to release new version of avrdude.

[mcuee]

I will close this issue as #1162 will fix this issue and it is more generic (Linux, Windows and macOS static link binaries).

• To add arduino avrdude-packing github action to create binary for Linux, macOS and Windows #1162

[mcuee]

Similar discussion here.

• AVRDUDE binary to be used for projects #1213

[mcuee]

As of now, avrdude project will only release Windows binary for avrdude and not Linux/macOS binary.

If you need Linux (or macOS) static link avrdude binary for production purpose, it is good to use the release
from Arduino-packaging project.
https://github.com/arduino/avrdude-packing/releases

For testing purpose, you can try my snapshot release here.
https://github.com/mcuee/avrdude-packing/releases

[mcuee]

Re-open the issue but I make it more generic.

[mcuee]

Potential solutions but need more work on CMake.

• To add arduino avrdude-packing github action to create binary for Linux, macOS and Windows #1162
• Enhance CMake to be able to support static link for Linux and macOS #1159

Related discussions:

• AVRDUDE binary to be used for projects #1213

[mcuee]

Even though #1162 can be a potential solution, right now at least two patches used by @umbynos are not really acceptable for the avrdude project.

Patch 06 may need some work. Patch 07 is just a hack.
We need CMake experts like @mariusgreuel to come out with the real solution.

https://github.com/arduino/avrdude-packing/tree/main/patches
Patch 06: https://github.com/arduino/avrdude-packing/blob/main/patches/0006-Cmake-add-libtinfo-and-libncurses-as-dependencies.patch
Patch 07: https://github.com/arduino/avrdude-packing/blob/main/patches/0007-Cmake-fix-build-not-working-with-mingw-toolchain.patch

(Patch 08 is just Arduino branding)

[mcuee]

After some thoughts, I have closed #1162. I think we should get #1159 done first and then this one.

This is not a high priority either since there is the Arduino avrdude-packing project out there which provides the infrastructrure to build the static link version for Linux/macOS and Windows.

Majority of the Linux users will still get avrdude through the Linux distro or build from source (typically with dynamic linking to dependacies). Majority of the macOS users will get avrdude through Homebrew or macPorts.

[mcuee]

@kristofmulier

You can use this one from Arduino avrdude-packing project.
https://github.com/arduino/avrdude-packing/releases/tag/7.1-arduino.1

[mcuee]

I will close this as not planned as the project has no intention to publish official Linux and macOS binaries.

[mcuee]

Converted this to conversation so that people can still discuss this issue.

[mcuee]

Arduino avrdude-packing project has released their version of Arduino 7.2, which should be suitable for the usage here.
https://github.com/arduino/avrdude-packing/releases/tag/7.2-arduino.1

[mcuee]

For those who want to try git main, I have some snapshot release build here.
https://github.com/mcuee/avrdude/releases

Latest version is here:
https://github.com/mcuee/avrdude/releases/tag/snapshot_19Oct2023

[stefanrueger]

Wonderful @mcuee. Can I suggest naming the snapshots in yyyy-mm-dd format (ISO)? This means that alphabetic order coincides with time order, and it's easier to find the newest one in your directory. In bash you can use snapshot_$(date -I)

[mcuee]

I see. I will follow the new tag name format from now on.

For the existing tags, I am not so sure if I can do anything about them without messing up the existing release.
https://github.com/mcuee/avrdude/tags

[mcuee]

Done for the latest one and the previous one. I also deleted the old ones since they are not useful now.
https://github.com/mcuee/avrdude/tags
https://github.com/mcuee/avrdude/releases/tag/snapshot_2023-10-19
https://github.com/mcuee/avrdude/releases/tag/snapshot_2023-07-15

I am following the guide here.
https://phoenixnap.com/kb/git-rename-tag

[mcuee]

Latest snapshot:
https://github.com/mcuee/avrdude/releases/tag/snapshot_2023-11-03

[mcuee]

PR #1540 has been merged. I think this can be closed.

• Add Arduino avrdude_packing github action for git man #1540