Plugins does not work with statically linked binaries

[itouch5000]

What is your current rclone version (output from rclone version)?

rclone v1.53.1
- os/arch: linux/amd64
- go version: go1.15

What problem are you are trying to solve?

Plugins are not usable with the precompiled rclone release because its statically linked. Plugins requires a dynamically linked rclone release.

How do you think rclone should be changed to solve that?

Please provide an additionally dynamically linked release or provide a dynamically linked release instead of a statically linked one if this should have no negative effects.

Forum link: https://forum.rclone.org/t/plugin-not-implemented/19886/4

[ncw]

A dynamic build would also have the effect that we could supply rclone cmount too which would be advantageous as it is more nearly a standard FUSE filing system.

I'm not sure whether changing from a static(ish) build to a dynamic build will break things for users.

The current build process produces this

$ go run bin/cross-compile.go -compile-only -include linux/amd64 v1.54-DEV
$ ldd build/rclone-v1.54-DEV-linux-amd64/rclone 
	not a dynamic executable

Whereas a cgo build produces this

$ go run bin/cross-compile.go -compile-only -include linux/amd64 -tags cmount -cgo v1.54-DEV
$ ldd build/rclone-v1.54-DEV-linux-amd64/rclone 
	linux-vdso.so.1 (0x00007ffd7a1ce000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdcee1a3000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdcee180000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdcedf8e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fdcee1c6000)

The safe thing to do would be to make a separate cgo build for linux/amd64. We actually build this exact executable in the linux build but we don't store it anywhere.

It is difficult to build cgo builds for any architectures we don't have build machines for so making anything other than the linux/amd64 build would be hard. This might involve using xgo...

Thoughts?

[itouch5000]

The current static build has no dependencies. It's perfect for example for alpine docker containers.

$ go run bin/cross-compile.go -compile-only -include linux/amd64 v1.54-DEV
2020/10/29 22:11:37 Compiling linux/amd64 into rclone-v1.54-DEV-linux-amd64
2020/10/29 22:11:40 Done compiling linux/amd64
2020/10/29 22:11:40 Compiled 1 arches in 2.518863957s

$ docker run -it --rm -v $(pwd)/build/rclone-v1.54-DEV-linux-amd64:/usr/local/bin alpine:latest /bin/sh
/ # rclone --version
rclone v1.54-DEV
- os/arch: linux/amd64
- go version: go1.15.3

---

The dynamic cgo build does not work as above because its compiled against glibc. For example alpine does contain glibc, a manual installation is required.

$ go run bin/cross-compile.go -compile-only -include linux/amd64 -tags cmount -cgo v1.54-DEV
2020/10/29 22:12:40 Compiling linux/amd64 into rclone-v1.54-DEV-linux-amd64
2020/10/29 22:12:44 Done compiling linux/amd64
2020/10/29 22:12:44 Compiled 1 arches in 4.633763348s

$ docker run -it --rm -v $(pwd)/build/rclone-v1.54-DEV-linux-amd64:/usr/local/bin alpine:latest /bin/sh
/ # rclone --version
/bin/sh: rclone: not found
/ # ls -la /usr/local/bin
total 45280
drwxr-xr-x    2 1000     985             60 Oct 29 21:12 .
drwxr-xr-x    5 root     root            41 Oct 21 09:23 ..
-rwxr-xr-x    1 1000     985       46365472 Oct 29 21:12 rclone
/ # echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/ # wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
/ # wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.32-r0/glibc-2.32-r0.apk
Connecting to github.com (140.82.121.3:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.113.131:443)
saving to 'glibc-2.32-r0.apk'
glibc-2.32-r0.apk    100% |**********************************************************************************************************************************************| 4331k  0:00:00 ETA
'glibc-2.32-r0.apk' saved
/ # apk add glibc-2.32-r0.apk
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/1) Installing glibc (2.32-r0)
OK: 15 MiB in 15 packages
/ # rclone --version
rclone v1.54-DEV
- os/arch: linux/amd64
- go version: go1.15.3

So I suggest to provide a separate cgo build for linux/amd64.

[ivandeex]

Since linux distros have different versions of glibc (or even musl instead of glibc), rclone will have to pick one (or two) popular distros to provide cgo build for,
e.g. alpine (latest?) or debian (stretch) or ubuntu (bionic)..... or what?
It's a slippery slope, I deem

[itouch5000]

Or you use a very old version of glibc to build rclone, because it will run on all distributions with newer glibc versions?

[ivandeex]

It will still need CI tests on supported distros.

[ncw]

Here is another idea for cross building dynamic libraries: https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho

zig can act as a C toolchain and out of the box can cross compile for many platforms.

[ivandeex]

Related to #5090

[ivandeex]

closing in favor of #5090