Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storj broken on ARMv7 since 1.64.0: unaligned 64-bit atomic operation #7339

Closed
N-Storm opened this issue Sep 25, 2023 · 7 comments
Closed

Storj broken on ARMv7 since 1.64.0: unaligned 64-bit atomic operation #7339

N-Storm opened this issue Sep 25, 2023 · 7 comments
Labels
bug Remote: Storj
Milestone
v1.66

Comments

@N-Storm
Copy link

N-Storm commented Sep 25, 2023

The associated forum post URL from https://forum.rclone.org

What is the problem you are having with rclone?

I'm running rclone on OrangePi One:

processor       : 3
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 22.85
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : Allwinner sun8i Family
Revision        : 0000
Serial          : 02c0008146808f5b

Was working great so far on 1.63.1 and earlier versions. With 1.64.0 (official builds, both rclone-v1.64.0-linux-arm-v7.deb package & binary from rclone selfupdate are affected) I'm getting the following error with Storj storage only on transfers (i.e. listing with lsl still works):
panic: unaligned 64-bit atomic operation

storj-raw is a remote configure as storj:

[storj-raw]
type = storj
access_grant = xxxxxxxxxxxxxxxxxxx

This remote works fine with the same config and rclone version, but on x86-64 host, so it's only ARM (v7) related.

What is your rclone version (output from rclone version)

rclone v1.63.1
- os/version: debian 12.1
- os/kernel: 6.5.1-edge-sunxi (armv7l)
- os/type: linux
- os/arch: arm (ARMv7 compatible)
- go/version: go1.20.6
- go/linking: static
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Armbian 23.11.0-trunk.41 Bookworm with bleeding edge Linux 6.5.1-edge-sunxi

Which cloud storage system are you using? (e.g. Google Drive)

Storj

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone copy LICENSE storj-raw:backups/ -vv

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2023/09/25 18:31:05 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "LICENSE" "storj-raw:backups/" "-vv"]
2023/09/25 18:31:05 DEBUG : Creating backend with remote "LICENSE"
2023/09/25 18:31:05 DEBUG : Using config file from "/home/nstorm/.config/rclone/rclone.conf"
2023/09/25 18:31:05 DEBUG : fs cache: adding new entry for parent of "LICENSE", "/home/nstorm"
2023/09/25 18:31:05 DEBUG : Creating backend with remote "storj-raw:backups/"
2023/09/25 18:31:05 DEBUG : FS sj://backups: connecting...
2023/09/25 18:31:05 DEBUG : FS sj://backups: connected: <nil>
2023/09/25 18:31:05 DEBUG : fs cache: renaming cache item "storj-raw:backups/" to be canonical "storj-raw:backups"
2023/09/25 18:31:05 DEBUG : FS sj://backups: stat ./LICENSE
2023/09/25 18:31:05 DEBUG : FS sj://backups: err: uplink: object not found ("LICENSE")
        storj.io/uplink.errwrapf:108
        storj.io/uplink.convertKnownErrors:59
        storj.io/uplink.(*Project).StatObject:93
        github.com/rclone/rclone/backend/storj.(*Fs).NewObject:514
        github.com/rclone/rclone/fs/operations.moveOrCopyFile:1982
        github.com/rclone/rclone/fs/operations.CopyFile:2076
        github.com/rclone/rclone/cmd/copy.glob..func1.1:97
        github.com/rclone/rclone/cmd.Run:258
        github.com/rclone/rclone/cmd/copy.glob..func1:93
        github.com/spf13/cobra.(*Command).execute:944
        github.com/spf13/cobra.(*Command).ExecuteC:1068
        github.com/spf13/cobra.(*Command).Execute:992
        github.com/rclone/rclone/cmd.Main:570
        main.main:14
        runtime.main:267
2023/09/25 18:31:05 DEBUG : LICENSE: Need to transfer - File not found at Destination
2023/09/25 18:31:05 DEBUG : FS sj://backups: cp input ./LICENSE # [HashesOption([])] 7048
panic: unaligned 64-bit atomic operation

goroutine 1 [running]:
runtime/internal/atomic.panicUnaligned()
        runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Load64(0x40ae2dc)
        runtime/internal/atomic/atomic_arm.s:280 +0x14
storj.io/uplink/private/storage/streams/buffer.(*MemoryBackend).Write(0x40ae2d0, {0x447c000, 0x1000, 0x1d00})
        storj.io/uplink@v1.11.0/private/storage/streams/buffer/backend.go:38 +0x30
storj.io/uplink/private/storage/streams/buffer.(*Buffer).Write(0x3e80420, {0x447c000, 0x1d00, 0x1d00})
        storj.io/uplink@v1.11.0/private/storage/streams/buffer/buffer.go:63 +0x140
storj.io/common/encryption.(*transformedWriter).Write(0x443c1e0, {0x84ac000, 0x168, 0x168})
        storj.io/common@v0.0.0-20230602145716-d6ea82d58b3d/encryption/transform.go:237 +0x270
storj.io/common/encryption.(*transformedWriter).Close(0x443c1e0)
        storj.io/common@v0.0.0-20230602145716-d6ea82d58b3d/encryption/transform.go:255 +0x74
storj.io/uplink/private/storage/streams/splitter.(*encryptedBuffer).DoneWriting(0x3e80440, {0x0, 0x0})
        storj.io/uplink@v1.11.0/private/storage/streams/splitter/encrypted_buffer.go:53 +0x80
storj.io/uplink/private/storage/streams/splitter.(*Splitter).Finish.(*baseSplitter).Finish.func1()
        storj.io/uplink@v1.11.0/private/storage/streams/splitter/base_splitter.go:57 +0xa0
sync.(*Once).doSlow(0x4348188, 0x84ab770)
        sync/once.go:74 +0xb8
sync.(*Once).Do(0x4348188, 0x84ab770)
        sync/once.go:65 +0x40
storj.io/uplink/private/storage/streams/splitter.(*baseSplitter).Finish(...)
        storj.io/uplink@v1.11.0/private/storage/streams/splitter/base_splitter.go:52
storj.io/uplink/private/storage/streams/splitter.(*Splitter).Finish(0x4040700, {0x0, 0x0})
        storj.io/uplink@v1.11.0/private/storage/streams/splitter/splitter.go:117 +0x68
storj.io/uplink/private/storage/streams.(*Upload).Commit(0x4262540)
        storj.io/uplink@v1.11.0/private/storage/streams/upload.go:66 +0x48
storj.io/uplink.(*Upload).Commit(0x3d22e80)
        storj.io/uplink@v1.11.0/upload.go:196 +0xb8
github.com/rclone/rclone/backend/storj.(*Fs).put(0x40a90c0, {0x2227608, 0x31f9700}, {0x2214a5c, 0x40e42c0}, {0xae62f538, 0x40a96c0}, {0xbefaa665, 0x7}, {0x3c52d58, ...})
        github.com/rclone/rclone/backend/storj/fs.go:592 +0x6d0
github.com/rclone/rclone/backend/storj.(*Fs).Put(0x40a90c0, {0x2227608, 0x31f9700}, {0x2214a5c, 0x40e42c0}, {0xae62f538, 0x40a96c0}, {0x3c52d58, 0x1, 0x1})
        github.com/rclone/rclone/backend/storj/fs.go:537 +0x88
github.com/rclone/rclone/fs/operations.Copy({0x2227608, 0x31f9700}, {0x2231b80, 0x40a90c0}, {0x0, 0x0}, {0xbefaa665, 0x7}, {0x223116c, 0x40a96c0})
        github.com/rclone/rclone/fs/operations/operations.go:476 +0x2294
github.com/rclone/rclone/fs/operations.moveOrCopyFile({0x2227608, 0x31f9700}, {0x2231b80, 0x40a90c0}, {0x2231130, 0x3dfaf50}, {0xbefaa665, 0x7}, {0xbefaa665, 0x7}, ...)
        github.com/rclone/rclone/fs/operations/operations.go:2056 +0xa00
github.com/rclone/rclone/fs/operations.CopyFile(...)
        github.com/rclone/rclone/fs/operations/operations.go:2076
github.com/rclone/rclone/cmd/copy.glob..func1.1()
        github.com/rclone/rclone/cmd/copy/copy.go:97 +0xc4
github.com/rclone/rclone/cmd.Run(0x1, 0x1, 0x31c1d68, 0x84abe4c)
        github.com/rclone/rclone/cmd/cmd.go:258 +0xf8
github.com/rclone/rclone/cmd/copy.glob..func1(0x31c1d68, {0x43af290, 0x2, 0x3})
        github.com/rclone/rclone/cmd/copy/copy.go:93 +0xc8
github.com/spf13/cobra.(*Command).execute(0x31c1d68, {0x43af278, 0x3, 0x3})
        github.com/spf13/cobra@v1.7.0/command.go:944 +0x780
github.com/spf13/cobra.(*Command).ExecuteC(0x31bf248)
        github.com/spf13/cobra@v1.7.0/command.go:1068 +0x410
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.7.0/command.go:992
github.com/rclone/rclone/cmd.Main()
        github.com/rclone/rclone/cmd/cmd.go:570 +0x98
main.main()
        github.com/rclone/rclone/rclone.go:14 +0x14

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@N-Storm
Copy link
Author

N-Storm commented Sep 25, 2023

Downgrading to 1.63.1 from https://downloads.rclone.org/v1.63.1/ works as workaround for now.

@ncw
Copy link
Member

ncw commented Sep 27, 2023

This looks like a bug in the Storj code.

storj.io/uplink/private/storage/streams/buffer.(*MemoryBackend).Write(0x40ae2d0, {0x447c000, 0x1000, 0x1d00})

storj.io/uplink@v1.11.0/private/storage/streams/buffer/backend

It would be worth checking their GitHub to see if the know about it or have fixed it already

@solracsf solracsf mentioned this issue Sep 27, 2023
Closed
@N-Storm
Copy link
Author

N-Storm commented Oct 17, 2023

This looks like a bug in the Storj code.

storj.io/uplink/private/storage/streams/buffer.(*MemoryBackend).Write(0x40ae2d0, {0x447c000, 0x1000, 0x1d00})

storj.io/uplink@v1.11.0/private/storage/streams/buffer/backend

It would be worth checking their GitHub to see if the know about it or have fixed it already

Seems like this one are related: storj/uplink#156 and it was fixed in v1.12.1

@N-Storm
Copy link
Author

N-Storm commented Oct 17, 2023 •
edited

Ok, I've did a quick rclone build from sources with storj/uplink version bumped to v1.12.1 and can confirm this fixes the issue on ARM 32-bit for me.

  1. Got release tagged v1.64.1: https://github.com/rclone/rclone/archive/refs/tags/v1.64.1.zip
  2. Unzip and modified go.mod to update storj/uplink version to v1.12.1:
--- go.mod.orig 2023-10-17 22:16:22.979674046 +0300
+++ go.mod      2023-10-17 21:59:21.914461479 +0300
@@ -74,7 +74,7 @@
        golang.org/x/time v0.3.0
        google.golang.org/api v0.134.0
        gopkg.in/yaml.v2 v2.4.0
-       storj.io/uplink v1.12.0
+       storj.io/uplink v1.12.1
 )
  1. Updated checksums in go.sum as instructed by go: go get github.com/rclone/rclone/backend/storj
  2. Cross-compiled (on Debian 11 x86_64 host with arm-linux-gnueabihf toolchain installed):
 $ GOOS=linux GOARCH=arm CGO_ENABLED=0 go build && arm-linux-gnueabihf-strip rclone
 $ file rclone
rclone: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, Go BuildID=rPiauQilOvvmBBGHplnJ/9NvLXUW7bjLuLIoHy4r3/5TlC-VIotuACNueqgmIP/MkYFlnb_sXN2SQjXVbNP, stripped
  1. Uploaded resulting binary to my OrangePi One host and tested against pre-compiled version:
 $ wget https://downloads.rclone.org/v1.64.1/rclone-v1.64.1-linux-arm-v7.zip && unzip rclone-v1.64.1-linux-arm-v7.zip

 # Original pre-compiled binary
 $ rclone-v1.64.1-linux-arm-v7/rclone version
rclone v1.64.1
- os/version: debian 12.2
- os/kernel: 6.5.6-edge-sunxi (armv7l)
- os/type: linux
- os/arch: arm (ARMv7 compatible)
- go/version: go1.21.3
- go/linking: static
- go/tags: none

 # Updated build:
 $ ./rclone version
rclone v1.64.1-DEV
- os/version: debian 12.2
- os/kernel: 6.5.6-edge-sunxi (armv7l)
- os/type: linux
- os/arch: arm (ARMv7 compatible)
- go/version: go1.21.3
- go/linking: static
- go/tags: none

 # Original pre-compiled binary
 $ rclone-v1.64.1-linux-arm-v7/rclone copy ./rclone-v1.64.1-linux-arm-v7/README.txt storj-raw:backups/ -vv
2023/10/17 22:23:26 DEBUG : rclone: Version "v1.64.1" starting with parameters ["rclone-v1.64.1-linux-arm-v7/rclone" "copy" "./rclone-v1.64.1-linux-arm-v7/README.txt" "storj-raw:backups/" "-vv"]
2023/10/17 22:23:26 DEBUG : Creating backend with remote "./rclone-v1.64.1-linux-arm-v7/README.txt"
2023/10/17 22:23:26 DEBUG : Using config file from "/home/nstorm/.config/rclone/rclone.conf"
2023/10/17 22:23:26 DEBUG : fs cache: adding new entry for parent of "./rclone-v1.64.1-linux-arm-v7/README.txt", "/tmp/rclone/rclone-v1.64.1-linux-arm-v7"
2023/10/17 22:23:26 DEBUG : Creating backend with remote "storj-raw:backups/"
2023/10/17 22:23:26 DEBUG : FS sj://backups: connecting...
2023/10/17 22:23:26 DEBUG : FS sj://backups: connected: <nil>
2023/10/17 22:23:26 DEBUG : fs cache: renaming cache item "storj-raw:backups/" to be canonical "storj-raw:backups"
2023/10/17 22:23:26 DEBUG : FS sj://backups: stat ./README.txt
2023/10/17 22:23:26 DEBUG : FS sj://backups: err: uplink: object not found ("README.txt")
        storj.io/uplink.errwrapf:108
        storj.io/uplink.convertKnownErrors:59
        storj.io/uplink.(*Project).StatObject:93
        github.com/rclone/rclone/backend/storj.(*Fs).NewObject:511
        github.com/rclone/rclone/fs/operations.moveOrCopyFile:1984
        github.com/rclone/rclone/fs/operations.CopyFile:2078
        github.com/rclone/rclone/cmd/copy.glob..func1.1:97
        github.com/rclone/rclone/cmd.Run:258
        github.com/rclone/rclone/cmd/copy.glob..func1:93
        github.com/spf13/cobra.(*Command).execute:944
        github.com/spf13/cobra.(*Command).ExecuteC:1068
        github.com/spf13/cobra.(*Command).Execute:992
        github.com/rclone/rclone/cmd.Main:570
        main.main:14
        runtime.main:267
2023/10/17 22:23:26 DEBUG : README.txt: Need to transfer - File not found at Destination
2023/10/17 22:23:26 DEBUG : FS sj://backups: cp input ./README.txt # [HashesOption([])] 1996824
panic: unaligned 64-bit atomic operation

goroutine 1 [running]:
runtime/internal/atomic.panicUnaligned()
        runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Load64(0x49beb64)
        runtime/internal/atomic/atomic_arm.s:280 +0x14
storj.io/uplink/private/storage/streams/buffer.(*MemoryBackend).Write(0x49beb58, {0x4c12000, 0x1000, 0x1d00})
        storj.io/uplink@v1.12.0/private/storage/streams/buffer/backend.go:38 +0x30
storj.io/uplink/private/storage/streams/buffer.(*Buffer).Write(0x4899a20, {0x4c12000, 0x1d00, 0x1d00})
        storj.io/uplink@v1.12.0/private/storage/streams/buffer/buffer.go:63 +0x140
 # ... etc, like in original issue description
 
 # Updated build:
 $ ./rclone copy ./rclone-v1.64.1-linux-arm-v7/README.txt storj-raw:backups/ -vv
2023/10/17 22:26:14 DEBUG : rclone: Version "v1.64.1-DEV" starting with parameters ["./rclone" "copy" "./rclone-v1.64.1-linux-arm-v7/README.txt" "storj-raw:backups/" "-vv"]
2023/10/17 22:26:14 DEBUG : Creating backend with remote "./rclone-v1.64.1-linux-arm-v7/README.txt"
2023/10/17 22:26:14 DEBUG : Using config file from "/home/nstorm/.config/rclone/rclone.conf"
2023/10/17 22:26:14 DEBUG : fs cache: adding new entry for parent of "./rclone-v1.64.1-linux-arm-v7/README.txt", "/tmp/rclone/rclone-v1.64.1-linux-arm-v7"
2023/10/17 22:26:14 DEBUG : Creating backend with remote "storj-raw:backups/"
2023/10/17 22:26:14 DEBUG : FS sj://backups: connecting...
2023/10/17 22:26:14 DEBUG : FS sj://backups: connected: <nil>
2023/10/17 22:26:14 DEBUG : fs cache: renaming cache item "storj-raw:backups/" to be canonical "storj-raw:backups"
2023/10/17 22:26:14 DEBUG : FS sj://backups: stat ./README.txt
2023/10/17 22:26:14 DEBUG : FS sj://backups: err: uplink: object not found ("README.txt")
        storj.io/uplink.errwrapf:108
        storj.io/uplink.convertKnownErrors:59
        storj.io/uplink.(*Project).StatObject:93
        github.com/rclone/rclone/backend/storj.(*Fs).NewObject:511
        github.com/rclone/rclone/fs/operations.moveOrCopyFile:1984
        github.com/rclone/rclone/fs/operations.CopyFile:2078
        github.com/rclone/rclone/cmd/copy.glob..func1.1:97
        github.com/rclone/rclone/cmd.Run:258
        github.com/rclone/rclone/cmd/copy.glob..func1:93
        github.com/spf13/cobra.(*Command).execute:944
        github.com/spf13/cobra.(*Command).ExecuteC:1068
        github.com/spf13/cobra.(*Command).Execute:992
        github.com/rclone/rclone/cmd.Main:570
        main.main:14
        runtime.main:267
2023/10/17 22:26:14 DEBUG : README.txt: Need to transfer - File not found at Destination
2023/10/17 22:26:14 DEBUG : FS sj://backups: cp input ./README.txt # [HashesOption([])] 1996824
2023/10/17 22:26:18 INFO  : README.txt: Copied (new)
2023/10/17 22:26:18 INFO  :
Transferred:        1.904 MiB / 1.904 MiB, 100%, 485.740 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         4.8s

2023/10/17 22:26:18 DEBUG : 190 go routines active

Please bump storj/uplink version to v1.21.1 v1.12.1 to fix this issue in future rclone builds.

ncw added a commit that referenced this issue Oct 19, 2023
@ncw
storj: fix "unaligned 64-bit atomic operation" on 32 bit systems - fixes
71898cd 
 #7339

See: storj/uplink#156
Fixes: #7339
@ncw
Copy link
Member

ncw commented Oct 19, 2023

Can you try this please

v1.65.0-beta.7449.71898cdfe.fix-7339-storj-32bit on branch fix-7339-storj-32bit (uploaded in 15-30 mins)

@ncw ncw added the bug label Oct 19, 2023
@ncw ncw added this to the v1.65 milestone Oct 19, 2023
@N-Storm
Copy link
Author

N-Storm commented Oct 24, 2023

Yes, I could confirm this fixes the issue & the test build works as expected at least with copy operation:

$ ./rclone version && ./rclone copy ./test.dat storj-raw:hel-backups/ -vv
rclone v1.65.0-beta.7449.71898cdfe.fix-7339-storj-32bit
- os/version: debian 12.2
- os/kernel: 6.5.7-edge-sunxi (armv7l)
- os/type: linux
- os/arch: arm (ARMv7 compatible)
- go/version: go1.21.3
- go/linking: static
- go/tags: none
2023/10/24 23:49:34 DEBUG : rclone: Version "v1.65.0-beta.7449.71898cdfe.fix-7339-storj-32bit" starting with parameters ["./rclone" "copy" "./test.dat" "storj-raw:backups/" "-vv"]
2023/10/24 23:49:34 DEBUG : Creating backend with remote "./test.dat"
2023/10/24 23:49:34 DEBUG : Using config file from "/home/nstorm/.config/rclone/rclone.conf"
2023/10/24 23:49:34 DEBUG : fs cache: adding new entry for parent of "./test.dat", "/tmp"
2023/10/24 23:49:34 DEBUG : Creating backend with remote "storj-raw:backups/"
2023/10/24 23:49:34 DEBUG : FS sj://backups: connecting...
2023/10/24 23:49:34 DEBUG : FS sj://backups: connected: <nil>
2023/10/24 23:49:34 DEBUG : fs cache: renaming cache item "storj-raw:backups/" to be canonical "storj-raw:hel-backups"
2023/10/24 23:49:34 DEBUG : FS sj://backups: stat ./test.dat
2023/10/24 23:49:35 DEBUG : FS sj://backups: err: uplink: object not found ("test.dat")
        storj.io/uplink.errwrapf:108
        storj.io/uplink.convertKnownErrors:59
        storj.io/uplink.(*Project).StatObject:93
        github.com/rclone/rclone/backend/storj.(*Fs).NewObject:511
        github.com/rclone/rclone/fs/operations.moveOrCopyFile:1988
        github.com/rclone/rclone/fs/operations.CopyFile:2082
        github.com/rclone/rclone/cmd/copy.glob..func1.1:97
        github.com/rclone/rclone/cmd.Run:258
        github.com/rclone/rclone/cmd/copy.glob..func1:93
        github.com/spf13/cobra.(*Command).execute:944
        github.com/spf13/cobra.(*Command).ExecuteC:1068
        github.com/spf13/cobra.(*Command).Execute:992
        github.com/rclone/rclone/cmd.Main:570
        main.main:14
        runtime.main:267
2023/10/24 23:49:35 DEBUG : test.dat: Need to transfer - File not found at Destination
2023/10/24 23:49:35 DEBUG : FS sj://backups: cp input ./test.dat # [HashesOption([])] 16777216
2023/10/24 23:49:46 INFO  : test.dat: Copied (new)
2023/10/24 23:49:46 INFO  :
Transferred:           16 MiB / 16 MiB, 100%, 1.456 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        12.4s

2023/10/24 23:49:46 DEBUG : 202 go routines active

@ncw
Copy link
Member

ncw commented Oct 30, 2023

This should be fixed in the latest beta now and will be released in v1.65

@ncw ncw modified the milestones: v1.65, v1.66 Jan 3, 2024
@ncw ncw closed this as completed Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Remote: Storj
Projects
None yet
Milestone
v1.66
Development

No branches or pull requests
3 participants
@ncw @Animosity022 @N-Storm