Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

restic stalling on /etc under WSL1 #4730

Open
scy opened this issue Mar 16, 2024 · 2 comments
Open

restic stalling on /etc under WSL1 #4730

scy opened this issue Mar 16, 2024 · 2 comments

Comments

@scy
Copy link

scy commented Mar 16, 2024

Output of restic version

restic 0.16.4 compiled with go1.21.6 on linux/amd64

What backend/service did you use to store the repository?

rclone over SSH, with the remote running rclone serve restic --stdio, but it doesn't work via SFTP or using a local repo either.

Problem description / Steps to reproduce

Running under WSL1 in Windows 10, restic snapshot apparently stops reading files very early in the backup, unless I exclude /dev. Terminal output will still update, but other than that, all activity seems to stall. CPU usage stays at 0 to 3 %, memory usage stays constant. The issue occurs on local and remote repositories, and adding --exclude /dev fixes it.

GIF of the behavior

My command line (as root) is restic backup --dry-run --compression=auto --exclude-caches --exclude-file=/etc/restic/default.exclude --exclude-file=/etc/restic/network.exclude --exclude-file=/etc/restic/scy.exclude --password-file=/etc/restic/REDACTED.key --repo=sftp:HOST_REDACTED:PATH_REDACTED --tag=profile:system --verbose=2 /, usually launched via resticprofile, but the issue occurs also when running standalone. And yes, this is a dry run, but it also occurs in a real run.

Actually I don't mind simply excluding /dev as a workaround, but @fd0 asked me on Mastodon to create a bug report with a stack trace, so here we are 😉 I'll have to post the stack trace in a follow-up comment though because else this text is over 64 KiB and GitHub won't let me post it …

Expected behavior

restic creates a backup.

Actual behavior

restic stops creating a backup and apparently does nothing except for updating the terminal.

Do you have any idea what may have caused this?

WSL1 has all sorts of quirks; there's a reason it's no longer supported by Microsoft.

I assume there's something special in /dev that restic chokes on. Is there a way for me to see the path where it gets stuck? Note that it stops after three files in /bin (there are more than three in the directory 😉), so it looks to me as if not actually reading/chunking the files is the issue, but listing them.

Indeed, when I add --no-scan, it stalls later:

[…snip…]
new       /bin/busybox, saved in 0.152s (1.967 MiB added)
new       /bin/tar, saved in 0.028s (413.391 KiB added)
new       /bin/udevadm, saved in 0.031s (570.383 KiB added)
new       /boot/grub/gfxblacklist.txt, saved in 0.001s (712 B added)
new       /bin/zsh, saved in 0.032s (812.906 KiB added)
new       /bin/, saved in 0.260s (0 B added, 0 B stored, 10.188 KiB metadata)
new       /boot/config-5.4.0-1020-ibm, saved in 0.035s (232.298 KiB added)
new       /boot/System.map-5.4.0-1020-ibm, saved in 0.139s (4.381 MiB added)
new       /boot/grub/unicode.pf2, saved in 0.140s (2.286 MiB added)
new       /boot/grub/, saved in 0.142s (0 B added, 0 B stored, 439 B metadata)
new       /dev/block/, saved in 0.001s (0 B added, 0 B stored, 95 B metadata)
new       /boot/vmlinuz-5.4.0-1020-ibm, saved in 0.304s (10.449 MiB added)
new       /boot/initrd.img-5.4.0-1020-ibm, saved in 0.386s (20.326 MiB added)
new       /boot/, saved in 0.412s (0 B added, 0 B stored, 1.544 KiB metadata)
[0:24] 154 files, 53.169 MiB, 0 errors

Looks as if files are being read slightly out of order, for example /boot and /bin and /etc intermix with each other here, and this is with --read-concurrency 1.

Did restic help you today? Did it make you happy in any way?

Yes! restic is easy to install and configure while still being very powerful. I like the tags and ETA features and the flexibility that comes with using rclone as a destination.
@scy
Copy link
Author

scy commented Mar 16, 2024

Stack trace (including terminal output until `pkill -ABRT restic` 
open repository
repository c3a79bd2 opened (version 2, compression level auto)
lock repository
no parent snapshot found, will read all files
load index files
[0:00]          0 index files loaded
start scan on [/]
start backup on [/]
new       /bin/btrfs-debug-tree, saved in 0.088s (367.141 KiB added)
new       /bin/bash, saved in 0.098s (1.062 MiB added)
new       /bin/btrfs, saved in 0.097s (699.672 KiB added)
SIGABRT: abort 2.104 MiB, total 154 files 53.169 MiB, 0 errors
PC=0x46dcc3 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex()
        /usr/local/go/src/runtime/sys_linux_amd64.s:558 +0x23 fp=0x7ffff9329678 sp=0x7ffff9329670 pc=0x46dcc3
runtime.futexsleep(0x0?, 0xf93296f8?, 0x447ea7?)
        /usr/local/go/src/runtime/os_linux.go:69 +0x30 fp=0x7ffff93296c8 sp=0x7ffff9329678 pc=0x435510
runtime.notesleep(0x1b6c948)
        /usr/local/go/src/runtime/lock_futex.go:160 +0x87 fp=0x7ffff9329700 sp=0x7ffff93296c8 pc=0x40ea07
runtime.mPark(...)
        /usr/local/go/src/runtime/proc.go:1632
runtime.stopm()
        /usr/local/go/src/runtime/proc.go:2536 +0x8c fp=0x7ffff9329730 sp=0x7ffff9329700 pc=0x43fb2c
runtime.findRunnable()
        /usr/local/go/src/runtime/proc.go:3229 +0xb9c fp=0x7ffff9329840 sp=0x7ffff9329730 pc=0x44145c
runtime.schedule()
        /usr/local/go/src/runtime/proc.go:3582 +0xb1 fp=0x7ffff9329878 sp=0x7ffff9329840 pc=0x442251
runtime.park_m(0xc000007a00?)
        /usr/local/go/src/runtime/proc.go:3745 +0x11f fp=0x7ffff93298c0 sp=0x7ffff9329878 pc=0x44275f
traceback: unexpected SPWRITE function runtime.mcall
runtime.mcall()
        /usr/local/go/src/runtime/asm_amd64.s:458 +0x4e fp=0x7ffff93298d8 sp=0x7ffff93298c0 pc=0x46a02e

goroutine 1 [semacquire]:
runtime.gopark(0x1b9e500?, 0x40fade?, 0x40?, 0xf4?, 0xc0000061a0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000e0e80 sp=0xc0000e0e60 pc=0x43be2e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.semacquire1(0xc00012f910, 0x18?, 0x1, 0x0, 0x8b?)
        /usr/local/go/src/runtime/sema.go:160 +0x218 fp=0xc0000e0ee8 sp=0xc0000e0e80 pc=0x44c918
sync.runtime_Semacquire(0xc0000d8198?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc0000e0f20 sp=0xc0000e0ee8 pc=0x467fc5
sync.(*WaitGroup).Wait(0x11347e0?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x48 fp=0xc0000e0f48 sp=0xc0000e0f20 pc=0x478568
golang.org/x/sync/errgroup.(*Group).Wait(0xc00012f900)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:53 +0x25 fp=0xc0000e0f68 sp=0xc0000e0f48 pc=0x7708c5
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot(0xc0001d6090, {0x138ca08?, 0xc00016d590}, {0xc00041c050, 0x1, 0x1}, {{0xc0001100d0, 0x1, 0x1}, {0xc000425c1c, ...}, ...})
        /restic/internal/archiver/archiver.go:800 +0x332 fp=0xc0000e1120 sp=0xc0000e0f68 pc=0x7bd752
main.runBackup({_, _}, {{{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xc000050040, 0x3, ...}, ...}, ...}, ...)
        /restic/cmd/restic/cmd_backup.go:660 +0x16c8 fp=0xc0000e1860 sp=0xc0000e1120 pc=0xe74508
main.glob..func2(0xc000196800?, {0xc0000c6000, 0x1, 0xb})
        /restic/cmd/restic/cmd_backup.go:83 +0x508 fp=0xc0000e1c18 sp=0xc0000e1860 pc=0xe70468
github.com/spf13/cobra.(*Command).execute(0x1b583c0, {0xc0004f3290, 0xb, 0xb})
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x87c fp=0xc0000e1d50 sp=0xc0000e1c18 pc=0x76399c
github.com/spf13/cobra.(*Command).ExecuteC(0x1b5c5e0)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 fp=0xc0000e1e08 sp=0xc0000e1d50 pc=0x7641c5
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985
main.main()
        /restic/cmd/restic/main.go:109 +0x1a5 fp=0xc0000e1f40 sp=0xc0000e1e08 pc=0xeb4665
runtime.main()
        /usr/local/go/src/runtime/proc.go:267 +0x2bb fp=0xc0000e1fe0 sp=0xc0000e1f40 pc=0x43b9db
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000e1fe8 sp=0xc0000e1fe0 pc=0x46bea1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000072fa8 sp=0xc000072f88 pc=0x43be2e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000072fe0 sp=0xc000072fa8 pc=0x43bcb3
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000072fe8 sp=0xc000072fe0 pc=0x46bea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000073778 sp=0xc000073758 pc=0x43be2e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
        /usr/local/go/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000737c8 sp=0xc000073778 pc=0x427e1f
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000737e0 sp=0xc0000737c8 pc=0x41cee5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000737e8 sp=0xc0000737e0 pc=0x46bea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x93c10?, 0x3bb33400?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000073f70 sp=0xc000073f50 pc=0x43be2e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1b6b480)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000073fa0 sp=0xc000073f70 pc=0x4256a9
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000073fc8 sp=0xc000073fa0 pc=0x425c59
runtime.gcenable.func2()
        /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x41ce85
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x46bea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 17 [finalizer wait]:
runtime.gopark(0x40fade?, 0x400000?, 0x70?, 0x26?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000072620 sp=0xc000072600 pc=0x43be2e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000727e0 sp=0xc000072620 pc=0x41bf07
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000727e8 sp=0xc0000727e0 pc=0x46bea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 18 [select]:
runtime.gopark(0xc00006e788?, 0x3?, 0x88?, 0x1f?, 0xc00006e772?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006e618 sp=0xc00006e5f8 pc=0x43be2e
runtime.selectgo(0xc00006e788, 0xc00006e76c, 0xc00018cf80?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00006e738 sp=0xc00006e618 pc=0x44b8e5
go.opencensus.io/stats/view.(*worker).start(0xc00018cf80)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00006e7c8 sp=0xc00006e738 pc=0xd7e95f
go.opencensus.io/stats/view.init.0.func1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00006e7e0 sp=0xc00006e7c8 pc=0xd7dc85
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x46bea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 19 [chan receive]:
runtime.gopark(0xc00006eee8?, 0x4bf3e5?, 0x1?, 0x25?, 0xc00006ef60?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006eed0 sp=0xc00006eeb0 pc=0x43be2e
runtime.chanrecv(0xc000386120, 0xc00006efa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc00006ef48 sp=0xc00006eed0 pc=0x4099ad
runtime.chanrecv2(0xc000386120?, 0xc00006efc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc00006ef70 sp=0xc00006ef48 pc=0x4095d2
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:361 +0x99 fp=0xc00006efe0 sp=0xc00006ef70 pc=0x7b2bf9
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x46bea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
        /restic/internal/restic/lock.go:358 +0x1a

goroutine 33 [select, locked to thread]:
runtime.gopark(0xc0003947a8?, 0x2?, 0x60?, 0x46?, 0xc0003947a4?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000394638 sp=0xc000394618 pc=0x43be2e
runtime.selectgo(0xc0003947a8, 0xc0003947a0, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000394758 sp=0xc000394638 pc=0x44b8e5
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal_unix.go:1014 +0x19f fp=0xc0003947e0 sp=0xc000394758 pc=0x46303f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003947e8 sp=0xc0003947e0 pc=0x46bea1
created by runtime.ensureSigM in goroutine 19
        /usr/local/go/src/runtime/signal_unix.go:997 +0xc8

goroutine 34 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
        /usr/local/go/src/runtime/lock_futex.go:236 +0x29 fp=0xc000394fa0 sp=0xc000394f68 pc=0x40ece9
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000394fc0 sp=0xc000394fa0 pc=0x468729
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000394fe0 sp=0xc000394fc0 pc=0x4bf713
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000394fe8 sp=0xc000394fe0 pc=0x46bea1
created by os/signal.Notify.func1.1 in goroutine 19
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 20 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006f698 sp=0xc00006f678 pc=0x43be2e
runtime.chanrecv(0xc00012d620, 0xc00006f778, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc00006f710 sp=0xc00006f698 pc=0x4099ad
runtime.chanrecv2(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc00006f738 sp=0xc00006f710 pc=0x4095d2
main.CleanupHandler(0x0?)
        /restic/cmd/restic/cleanup.go:61 +0x38 fp=0xc00006f7c8 sp=0xc00006f738 pc=0xe6fc38
main.init.0.func1()
        /restic/cmd/restic/cleanup.go:21 +0x25 fp=0xc00006f7e0 sp=0xc00006f7c8 pc=0xe6f7e5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006f7e8 sp=0xc00006f7e0 pc=0x46bea1
created by main.init.0 in goroutine 1
        /restic/cmd/restic/cleanup.go:21 +0x89

goroutine 21 [GC worker (idle)]:
runtime.gopark(0xa907b8ef18c?, 0x3?, 0xf0?, 0x23?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00006ff50 sp=0xc00006ff30 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc00006ffe0 sp=0xc00006ff50 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 35 [GC worker (idle)]:
runtime.gopark(0xa90784f6c68?, 0x1?, 0xe8?, 0x99?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000395750 sp=0xc000395730 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003957e0 sp=0xc000395750 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003957e8 sp=0xc0003957e0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 36 [GC worker (idle)]:
runtime.gopark(0xa9077929250?, 0x3?, 0x9c?, 0xf3?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000395f50 sp=0xc000395f30 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000395fe0 sp=0xc000395f50 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000395fe8 sp=0xc000395fe0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 5 [GC worker (idle)]:
runtime.gopark(0xa90784f2a64?, 0x3?, 0x18?, 0xab?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000074750 sp=0xc000074730 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000747e0 sp=0xc000074750 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000747e8 sp=0xc0000747e0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x1b9e500?, 0x1?, 0x6c?, 0x66?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000396750 sp=0xc000396730 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0003967e0 sp=0xc000396750 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003967e8 sp=0xc0003967e0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x1b9e500?, 0x3?, 0x48?, 0x58?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000396f50 sp=0xc000396f30 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000396fe0 sp=0xc000396f50 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000396fe8 sp=0xc000396fe0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 6 [GC worker (idle)]:
runtime.gopark(0xa907907b304?, 0x1?, 0x74?, 0x6c?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000074f50 sp=0xc000074f30 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc000074fe0 sp=0xc000074f50 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 7 [GC worker (idle)]:
runtime.gopark(0xa907ac99a04?, 0x1?, 0x80?, 0xce?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000075750 sp=0xc000075730 pc=0x43be2e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000757e0 sp=0xc000075750 pc=0x41ea65
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000757e8 sp=0xc0000757e0 pc=0x46bea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1219 +0x1c

goroutine 8 [select]:
runtime.gopark(0xc000255f10?, 0x3?, 0x0?, 0x0?, 0xc000255eaa?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000255d20 sp=0xc000255d00 pc=0x43be2e
runtime.selectgo(0xc000255f10, 0xc000255ea4, 0x1?, 0x0, 0x3?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000255e40 sp=0xc000255d20 pc=0x44b8e5
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run(0xc00012df20, {0x138ca08, 0xc00016d9a0})
        /restic/internal/ui/termstatus/status.go:105 +0x119 fp=0xc000255f50 sp=0xc000255e40 pc=0x85bf39
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0x0?, {0x138ca08?, 0xc00016d9a0?})
        /restic/internal/ui/termstatus/status.go:94 +0x5d fp=0xc000255f98 sp=0xc000255f50 pc=0x85bd5d
main.glob..func2.2()
        /restic/cmd/restic/cmd_backup.go:72 +0x55 fp=0xc000255fe0 sp=0xc000255f98 pc=0xe705f5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000255fe8 sp=0xc000255fe0 pc=0x46bea1
created by main.glob..func2 in goroutine 1
        /restic/cmd/restic/cmd_backup.go:70 +0x1dc

goroutine 9 [IO wait]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x3?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003974a0 sp=0xc000397480 pc=0x43be2e
runtime.netpollblock(0x4819d8?, 0x407de6?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0003974d8 sp=0xc0003974a0 pc=0x4348d7
internal/poll.runtime_pollWait(0x7f5c0a738e58, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0003974f8 sp=0xc0003974d8 pc=0x466745
internal/poll.(*pollDesc).wait(0xc00007e300?, 0xc0004ff000?, 0x1)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000397520 sp=0xc0003974f8 pc=0x4a64a7
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00007e300, {0xc0004ff000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0003975b8 sp=0xc000397520 pc=0x4a779a
os.(*File).read(...)
        /usr/local/go/src/os/file_posix.go:29
os.(*File).Read(0xc000112108, {0xc0004ff000?, 0x0?, 0x0?})
        /usr/local/go/src/os/file.go:118 +0x52 fp=0xc0003975f8 sp=0xc0003975b8 pc=0x4b2f32
bufio.(*Scanner).Scan(0xc000397750)
        /usr/local/go/src/bufio/scan.go:214 +0x81b fp=0xc0003976d0 sp=0xc0003975f8 pc=0x675efb
github.com/restic/restic/internal/backend/sftp.startClient.func1()
        /restic/internal/backend/sftp/sftp.go:71 +0x152 fp=0xc0003977e0 sp=0xc0003976d0 pc=0xe4a6d2
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003977e8 sp=0xc0003977e0 pc=0x46bea1
created by github.com/restic/restic/internal/backend/sftp.startClient in goroutine 1
        /restic/internal/backend/sftp/sftp.go:69 +0x1ee

goroutine 10 [syscall]:
syscall.Syscall6(0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/syscall/syscall_linux.go:91 +0x30 fp=0xc000397dd0 sp=0xc000397d48 pc=0x483b10
os.(*Process).blockUntilWaitable(0xc000138090)
        /usr/local/go/src/os/wait_waitid.go:32 +0x76 fp=0xc000397ea8 sp=0xc000397dd0 pc=0x4b9976
os.(*Process).wait(0xc000138090)
        /usr/local/go/src/os/exec_unix.go:22 +0x25 fp=0xc000397f08 sp=0xc000397ea8 pc=0x4b27a5
os.(*Process).Wait(...)
        /usr/local/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc0000402c0)
        /usr/local/go/src/os/exec/exec.go:890 +0x45 fp=0xc000397f70 sp=0xc000397f08 pc=0x7c7c45
github.com/restic/restic/internal/backend/sftp.startClient.func2()
        /restic/internal/backend/sftp/sftp.go:97 +0x25 fp=0xc000397fe0 sp=0xc000397f70 pc=0xe4a4e5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000397fe8 sp=0xc000397fe0 pc=0x46bea1
created by github.com/restic/restic/internal/backend/sftp.startClient in goroutine 1
        /restic/internal/backend/sftp/sftp.go:96 +0x3b5

goroutine 11 [IO wait]:
runtime.gopark(0xc000075c68?, 0xb?, 0x0?, 0x0?, 0xa?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000075c88 sp=0xc000075c68 pc=0x43be2e
runtime.netpollblock(0x4819d8?, 0x407de6?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc000075cc0 sp=0xc000075c88 pc=0x4348d7
internal/poll.runtime_pollWait(0x7f5c0a738a78, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc000075ce0 sp=0xc000075cc0 pc=0x466745
internal/poll.(*pollDesc).wait(0xc00007e480?, 0xc00007c078?, 0x1)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000075d08 sp=0xc000075ce0 pc=0x4a64a7
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00007e480, {0xc00007c078, 0x4, 0x4})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc000075da0 sp=0xc000075d08 pc=0x4a779a
os.(*File).read(...)
        /usr/local/go/src/os/file_posix.go:29
os.(*File).Read(0xc000112498, {0xc00007c078?, 0x416cf4?, 0x7f5c512d0f18?})
        /usr/local/go/src/os/file.go:118 +0x52 fp=0xc000075de0 sp=0xc000075da0 pc=0x4b2f32
github.com/pkg/sftp.(*conn).Read(0xc0000061a0?, {0xc00007c078?, 0xc000075e48?, 0x450609?})
        <autogenerated>:1 +0x29 fp=0xc000075e10 sp=0xc000075de0 pc=0xe48469
io.ReadAtLeast({0x1381ca0, 0xc0001d6000}, {0xc00007c078, 0x4, 0x4}, 0x4)
        /usr/local/go/src/io/io.go:335 +0x90 fp=0xc000075e58 sp=0xc000075e10 pc=0x4a0e10
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
github.com/pkg/sftp.recvPacket({0x1381ca0, 0xc0001d6000}, 0x0, 0x0?)
        /home/build/go/pkg/mod/github.com/pkg/sftp@v1.13.6/packet.go:277 +0x99 fp=0xc000075ed0 sp=0xc000075e58 pc=0xe42419
github.com/pkg/sftp.(*conn).recvPacket(...)
        /home/build/go/pkg/mod/github.com/pkg/sftp@v1.13.6/conn.go:25
github.com/pkg/sftp.(*clientConn).recv(0xc0001d6000)
        /home/build/go/pkg/mod/github.com/pkg/sftp@v1.13.6/conn.go:72 +0xd4 fp=0xc000075f90 sp=0xc000075ed0 pc=0xe3fbf4
github.com/pkg/sftp.NewClientPipe.func1()
        /home/build/go/pkg/mod/github.com/pkg/sftp@v1.13.6/client.go:242 +0x58 fp=0xc000075fe0 sp=0xc000075f90 pc=0xe35cb8
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=0x46bea1
created by github.com/pkg/sftp.NewClientPipe in goroutine 1
        /home/build/go/pkg/mod/github.com/pkg/sftp@v1.13.6/client.go:239 +0x368

goroutine 49 [select]:
runtime.gopark(0xc000019f78?, 0x3?, 0x60?, 0x93?, 0xc000019f0a?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000019db0 sp=0xc000019d90 pc=0x43be2e
runtime.selectgo(0xc000019f78, 0xc000019f04, 0x1b6af00?, 0x0, 0x5c97b7f5?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000019ed0 sp=0xc000019db0 pc=0x44b8e5
github.com/restic/restic/internal/ui/progress.(*Updater).run(0xc0004ac4c0)
        /restic/internal/ui/progress/updater.go:73 +0x17d fp=0xc000019fc8 sp=0xc000019ed0 pc=0x78f8fd
github.com/restic/restic/internal/ui/progress.NewUpdater.func1()
        /restic/internal/ui/progress/updater.go:40 +0x25 fp=0xc000019fe0 sp=0xc000019fc8 pc=0x78f685
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000019fe8 sp=0xc000019fe0 pc=0x46bea1
created by github.com/restic/restic/internal/ui/progress.NewUpdater in goroutine 1
        /restic/internal/ui/progress/updater.go:40 +0x15f

goroutine 92 [select]:
runtime.gopark(0xc00029df18?, 0x2?, 0xec?, 0x53?, 0xc00029de8c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00029dcf0 sp=0xc00029dcd0 pc=0x43be2e
runtime.selectgo(0xc00029df18, 0xc00029de88, 0x16ab7432c60da76c?, 0x0, 0x2e076?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00029de10 sp=0xc00029dcf0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc00029df48 sp=0xc00029de10 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc00029df78 sp=0xc00029df48 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc00029dfe0 sp=0xc00029df78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00029dfe8 sp=0xc00029dfe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 90 [select]:
runtime.gopark(0xc00022df18?, 0x2?, 0x0?, 0x0?, 0xc00022de8c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00022dcf0 sp=0xc00022dcd0 pc=0x43be2e
runtime.selectgo(0xc00022df18, 0xc00022de88, 0xb016385fd96a7275?, 0x0, 0xc963?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00022de10 sp=0xc00022dcf0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc00022df48 sp=0xc00022de10 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc00022df78 sp=0xc00022df48 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc00022dfe0 sp=0xc00022df78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00022dfe8 sp=0xc00022dfe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 89 [select]:
runtime.gopark(0xc000017f18?, 0x2?, 0x0?, 0x0?, 0xc000017e8c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000017cf0 sp=0xc000017cd0 pc=0x43be2e
runtime.selectgo(0xc000017f18, 0xc000017e88, 0xc257db6a65ac9cc7?, 0x0, 0x802fb?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000017e10 sp=0xc000017cf0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x2?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc000017f48 sp=0xc000017e10 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc000017f78 sp=0xc000017f48 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000017fe0 sp=0xc000017f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000017fe8 sp=0xc000017fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 88 [select]:
runtime.gopark(0xc000015f18?, 0x2?, 0x1?, 0x0?, 0xc000015e8c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000015cf0 sp=0xc000015cd0 pc=0x43be2e
runtime.selectgo(0xc000015f18, 0xc000015e88, 0x9726d61b97ca295d?, 0x0, 0x80e3a?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000015e10 sp=0xc000015cf0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x2?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc000015f48 sp=0xc000015e10 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc000015f78 sp=0xc000015f48 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000015fe0 sp=0xc000015f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000015fe8 sp=0xc000015fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 87 [sync.Mutex.Lock]:
runtime.gopark(0xc00007f0e0?, 0x4?, 0xa0?, 0x65?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00022f340 sp=0xc00022f320 pc=0x43be2e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.semacquire1(0xc00041c40c, 0x0?, 0x3, 0x1, 0xd8?)
        /usr/local/go/src/runtime/sema.go:160 +0x218 fp=0xc00022f3a8 sp=0xc00022f340 pc=0x44c918
sync.runtime_SemacquireMutex(0x0?, 0x78?, 0x3?)
        /usr/local/go/src/runtime/sema.go:77 +0x25 fp=0xc00022f3e0 sp=0xc00022f3a8 pc=0x468085
sync.(*Mutex).lockSlow(0xc00041c408)
        /usr/local/go/src/sync/mutex.go:171 +0x15d fp=0xc00022f430 sp=0xc00022f3e0 pc=0x476c5d
sync.(*Mutex).Lock(...)
        /usr/local/go/src/sync/mutex.go:90
main.(*rejectionCache).Lock(...)
        /restic/cmd/restic/exclude.go:31
main.isExcludedByFile({0xc000046078, 0xf69900?}, {0x11f8f4a, 0xc}, {0x11f8f57, 0x2b}, 0xc00041c400)
        /restic/cmd/restic/exclude.go:151 +0x133 fp=0xc00022f498 sp=0xc00022f430 pc=0xeab093
main.rejectIfPresent.func1({0xc000046078?, 0x14?})
        /restic/cmd/restic/exclude.go:133 +0x2c fp=0xc00022f4e0 sp=0xc00022f498 pc=0xeaaf2c
main.runBackup.func1({0xc000046078, 0x14})
        /restic/cmd/restic/cmd_backup.go:559 +0x62 fp=0xc00022f518 sp=0xc00022f4e0 pc=0xe74b02
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc0001d6090, {0x138ca08, 0xc000168140}, {0xc0000461c8, 0x14}, {0xc000046078, 0x14}, 0x0)
        /restic/internal/archiver/archiver.go:354 +0x18a fp=0xc00022f7b0 sp=0xc00022f518 pc=0x7b9f4a
github.com/restic/restic/internal/archiver.(*Archiver).SaveDir(0xc0001d6090, {0x138ca08, 0xc000168140}, {0xc0001264ec, 0x4}, {0xc0001261b8, 0x4}, {0x138e9c8, 0xc00003e410}, 0x0, ...)
        /restic/internal/archiver/archiver.go:253 +0x3de fp=0xc00022f8d8 sp=0xc00022f7b0 pc=0x7b97be
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc0001d6090, {0x138ca08, 0xc000168140}, {0xc0001264ec, 0x4}, {0xc0001261b8, 0x4}, 0x0)
        /restic/internal/archiver/archiver.go:465 +0xfea fp=0xc00022fb70 sp=0xc00022f8d8 pc=0x7badaa
github.com/restic/restic/internal/archiver.(*Archiver).SaveTree(0xc0001d6090, {0x138ca08, 0xc000168140}, {0x1377918, 0x1}, 0xc00012f8c0, 0x0, 0x2?)
        /restic/internal/archiver/archiver.go:588 +0x61b fp=0xc00022fe18 sp=0xc00022fb70 pc=0x7bbffb
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1.1()
        /restic/internal/archiver/archiver.go:765 +0x199 fp=0xc00022ff78 sp=0xc00022fe18 pc=0x7be019
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc00022ffe0 sp=0xc00022ff78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00022ffe8 sp=0xc00022ffe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 86
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 86 [semacquire]:
runtime.gopark(0x2?, 0x7f5c0a731ee0?, 0x20?, 0x1?, 0x410245?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00023ddc8 sp=0xc00023dda8 pc=0x43be2e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.semacquire1(0xc00012f990, 0x18?, 0x1, 0x0, 0x8b?)
        /usr/local/go/src/runtime/sema.go:160 +0x218 fp=0xc00023de30 sp=0xc00023ddc8 pc=0x44c918
sync.runtime_Semacquire(0xc0000d81b0?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc00023de68 sp=0xc00023de30 pc=0x467fc5
sync.(*WaitGroup).Wait(0x11346c0?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x48 fp=0xc00023de90 sp=0xc00023de68 pc=0x478568
golang.org/x/sync/errgroup.(*Group).Wait(0xc00012f980)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:53 +0x25 fp=0xc00023deb0 sp=0xc00023de90 pc=0x7708c5
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1()
        /restic/internal/archiver/archiver.go:790 +0x215 fp=0xc00023df78 sp=0xc00023deb0 pc=0x7bdd55
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc00023dfe0 sp=0xc00023df78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00023dfe8 sp=0xc00023dfe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 85 [semacquire]:
runtime.gopark(0x1?, 0xc000390768?, 0x0?, 0xd8?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000390678 sp=0xc000390658 pc=0x43be2e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.semacquire1(0xc00012f950, 0x0?, 0x1, 0x0, 0x0?)
        /usr/local/go/src/runtime/sema.go:160 +0x218 fp=0xc0003906e0 sp=0xc000390678 pc=0x44c918
sync.runtime_Semacquire(0x0?)
        /usr/local/go/src/runtime/sema.go:62 +0x25 fp=0xc000390718 sp=0xc0003906e0 pc=0x467fc5
sync.(*WaitGroup).Wait(0x0?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x48 fp=0xc000390740 sp=0xc000390718 pc=0x478568
golang.org/x/sync/errgroup.(*Group).Wait(0xc00012f940)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:53 +0x25 fp=0xc000390760 sp=0xc000390740 pc=0x7708c5
github.com/restic/restic/internal/repository.(*Repository).StartPackUploader.func1()
        /restic/internal/repository/repository.go:594 +0x17 fp=0xc000390778 sp=0xc000390760 pc=0x850e17
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc0003907e0 sp=0xc000390778 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003907e8 sp=0xc0003907e0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 84 [select]:
runtime.gopark(0xc000390f48?, 0x2?, 0xa6?, 0x6c?, 0xc000390f14?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000390db0 sp=0xc000390d90 pc=0x43be2e
runtime.selectgo(0xc000390f48, 0xc000390f10, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000390ed0 sp=0xc000390db0 pc=0x44b8e5
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xc5 fp=0xc000390f78 sp=0xc000390ed0 pc=0x84b7a5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000390fe0 sp=0xc000390f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000390fe8 sp=0xc000390fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 83 [select]:
runtime.gopark(0xc000391748?, 0x2?, 0xa6?, 0x6c?, 0xc000391714?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0003915b0 sp=0xc000391590 pc=0x43be2e
runtime.selectgo(0xc000391748, 0xc000391710, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0003916d0 sp=0xc0003915b0 pc=0x44b8e5
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xc5 fp=0xc000391778 sp=0xc0003916d0 pc=0x84b7a5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc0003917e0 sp=0xc000391778 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0003917e8 sp=0xc0003917e0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 82 [select]:
runtime.gopark(0xc000237f48?, 0x2?, 0xa6?, 0x6c?, 0xc000237f14?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000237db0 sp=0xc000237d90 pc=0x43be2e
runtime.selectgo(0xc000237f48, 0xc000237f10, 0x0?, 0x0, 0x6?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000237ed0 sp=0xc000237db0 pc=0x44b8e5
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xc5 fp=0xc000237f78 sp=0xc000237ed0 pc=0x84b7a5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000237fe0 sp=0xc000237f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000237fe8 sp=0xc000237fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 81 [select]:
runtime.gopark(0xc000238748?, 0x2?, 0xa6?, 0x6c?, 0xc000238714?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0002385b0 sp=0xc000238590 pc=0x43be2e
runtime.selectgo(0xc000238748, 0xc000238710, 0x7?, 0x0, 0xc000216c80?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0002386d0 sp=0xc0002385b0 pc=0x44b8e5
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xc5 fp=0xc000238778 sp=0xc0002386d0 pc=0x84b7a5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc0002387e0 sp=0xc000238778 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002387e8 sp=0xc0002387e0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 80 [select]:
runtime.gopark(0xc000238f48?, 0x2?, 0xa6?, 0x6c?, 0xc000238f14?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000238db0 sp=0xc000238d90 pc=0x43be2e
runtime.selectgo(0xc000238f48, 0xc000238f10, 0x0?, 0x0, 0x2?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000238ed0 sp=0xc000238db0 pc=0x44b8e5
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xc5 fp=0xc000238f78 sp=0xc000238ed0 pc=0x84b7a5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000238fe0 sp=0xc000238f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000238fe8 sp=0xc000238fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 79 [IO wait]:
runtime.gopark(0x423e17?, 0xb?, 0x0?, 0x0?, 0xc?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0000cf5b0 sp=0xc0000cf590 pc=0x43be2e
runtime.netpollblock(0x4819d8?, 0x407de6?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:564 +0xf7 fp=0xc0000cf5e8 sp=0xc0000cf5b0 pc=0x4348d7
internal/poll.runtime_pollWait(0x7f5c0a738d60, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85 fp=0xc0000cf608 sp=0xc0000cf5e8 pc=0x466745
internal/poll.(*pollDesc).wait(0xc000386720?, 0xc000044060?, 0x1)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000cf630 sp=0xc0000cf608 pc=0x4a64a7
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000386720, {0xc000044060, 0x2b, 0x2b})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0000cf6c8 sp=0xc0000cf630 pc=0x4a779a
os.(*File).read(...)
        /usr/local/go/src/os/file_posix.go:29
os.(*File).Read(0xc0001ce030, {0xc000044060?, 0xc0000cf740?, 0x450609?})
        /usr/local/go/src/os/file.go:118 +0x52 fp=0xc0000cf708 sp=0xc0000cf6c8 pc=0x4b2f32
io.ReadAtLeast({0x137f000, 0xc0001ce030}, {0xc000044060, 0x2b, 0x2b}, 0x2b)
        /usr/local/go/src/io/io.go:335 +0x90 fp=0xc0000cf750 sp=0xc0000cf708 pc=0x4a0e10
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
main.isDirExcludedByFile({0xc000425b10?, 0xc00010e420?}, {0x11f8f4a?, 0x9?}, {0x11f8f57, 0x2b})
        /restic/cmd/restic/exclude.go:190 +0x239 fp=0xc0000cf860 sp=0xc0000cf750 pc=0xeab539
main.isExcludedByFile({0xc000425b10, 0xc0000cf900?}, {0x11f8f4a, 0xc}, {0x11f8f57, 0x2b}, 0xc00041c400)
        /restic/cmd/restic/exclude.go:158 +0x205 fp=0xc0000cf8c8 sp=0xc0000cf860 pc=0xeab165
main.rejectIfPresent.func1({0xc000425b10?, 0xa?})
        /restic/cmd/restic/exclude.go:133 +0x2c fp=0xc0000cf910 sp=0xc0000cf8c8 pc=0xeaaf2c
main.runBackup.func1({0xc000425b10, 0xa})
        /restic/cmd/restic/cmd_backup.go:559 +0x62 fp=0xc0000cf948 sp=0xc0000cf910 pc=0xe74b02
github.com/restic/restic/internal/archiver.(*Scanner).scan(0xc0004d81b0, {0x138ca08, 0xc0000a80a0}, {0x8?, 0x3?, 0xd2?, 0x8?}, {0xc000425b10, 0xa})
        /restic/internal/archiver/scanner.go:107 +0x92 fp=0xc0000cfa28 sp=0xc0000cf948 pc=0x7c0d52
github.com/restic/restic/internal/archiver.(*Scanner).scan(0xc0004d81b0, {0x138ca08, 0xc0000a80a0}, {0x4?, 0x0?, 0x4?, 0xc000424270?}, {0xc0004259b8, 0x8})
        /restic/internal/archiver/scanner.go:134 +0x435 fp=0xc0000cfb08 sp=0xc0000cfa28 pc=0x7c10f5
github.com/restic/restic/internal/archiver.(*Scanner).scan(0xc0004d81b0, {0x138ca08, 0xc0000a80a0}, {0xf80f80?, 0xc0000e9001?, 0xc0000cfd00?, 0x7c29da?}, {0xc000424270, 0x4})
        /restic/internal/archiver/scanner.go:134 +0x435 fp=0xc0000cfbe8 sp=0xc0000cfb08 pc=0x7c10f5
github.com/restic/restic/internal/archiver.(*Scanner).scanTree(0xc0004d81b0, {0x138ca08, 0xc0000a80a0}, {0xc000366200?, 0x1b?, 0x9b6f2ec400000300?, 0xc000198000?}, {0x0, {0xc000424270, 0x4}, ...})
        /restic/internal/archiver/scanner.go:49 +0x10e fp=0xc0000cfd10 sp=0xc0000cfbe8 pc=0x7c06ce
github.com/restic/restic/internal/archiver.(*Scanner).scanTree(0xc0004d81b0, {0x138ca08, 0xc0000a80a0}, {0xc0002396d8?, 0x44cba5?, 0x1b74860?, 0xc0001d2440?}, {0xc00010e360, {0x0, 0x0}, ...})
        /restic/internal/archiver/scanner.go:60 +0x2dc fp=0xc0000cfe38 sp=0xc0000cfd10 pc=0x7c089c
github.com/restic/restic/internal/archiver.(*Scanner).Scan(0xc0004d81b0, {0x138ca08, 0xc0000a80a0}, {0xc00041c050, 0x1, 0x1})
        /restic/internal/archiver/scanner.go:91 +0x1e5 fp=0xc0000cff38 sp=0xc0000cfe38 pc=0x7c0b85
main.runBackup.func5()
        /restic/cmd/restic/cmd_backup.go:623 +0x2b fp=0xc0000cff78 sp=0xc0000cff38 pc=0xe749ab
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc0000cffe0 sp=0xc0000cff78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 91 [select]:
runtime.gopark(0xc0002a1f18?, 0x2?, 0x0?, 0x0?, 0xc0002a1e8c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0002a1cf0 sp=0xc0002a1cd0 pc=0x43be2e
runtime.selectgo(0xc0002a1f18, 0xc0002a1e88, 0x7528798d87c273f?, 0x0, 0x5bc90?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0002a1e10 sp=0xc0002a1cf0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x10?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc0002a1f48 sp=0xc0002a1e10 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc0002a1f78 sp=0xc0002a1f48 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc0002a1fe0 sp=0xc0002a1f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002a1fe8 sp=0xc0002a1fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 93 [select]:
runtime.gopark(0xc000236718?, 0x2?, 0x7?, 0x0?, 0xc00023668c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc0002364f0 sp=0xc0002364d0 pc=0x43be2e
runtime.selectgo(0xc000236718, 0xc000236688, 0x0?, 0x0, 0x2?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000236610 sp=0xc0002364f0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x2?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc000236748 sp=0xc000236610 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc000236778 sp=0xc000236748 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc0002367e0 sp=0xc000236778 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0002367e8 sp=0xc0002367e0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 94 [select]:
runtime.gopark(0xc00001bf18?, 0x2?, 0x2?, 0x0?, 0xc00001be8c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00001bcf0 sp=0xc00001bcd0 pc=0x43be2e
runtime.selectgo(0xc00001bf18, 0xc00001be88, 0xdd3fa1d8275374f9?, 0x0, 0x83142?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00001be10 sp=0xc00001bcf0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xa?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc00001bf48 sp=0xc00001be10 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc00001bf78 sp=0xc00001bf48 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc00001bfe0 sp=0xc00001bf78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00001bfe8 sp=0xc00001bfe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 95 [select]:
runtime.gopark(0xc000237718?, 0x2?, 0x5?, 0x0?, 0xc00023768c?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000225cf0 sp=0xc000225cd0 pc=0x43be2e
runtime.selectgo(0xc000225f18, 0xc000237688, 0x0?, 0x0, 0x6?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000225e10 sp=0xc000225cf0 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x6?, {0x138ca08, 0xc000168140}, 0xc00038c2a0)
        /restic/internal/archiver/blob_saver.go:86 +0xc5 fp=0xc000225f48 sp=0xc000225e10 pc=0x7be865
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:33 +0x25 fp=0xc000225f78 sp=0xc000225f48 pc=0x7be4c5
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000225fe0 sp=0xc000225f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000225fe8 sp=0xc000225fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 96 [select]:
runtime.gopark(0xc00029bdf0?, 0x2?, 0x40?, 0x46?, 0xc00029bdd4?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00029bc38 sp=0xc00029bc18 pc=0x43be2e
runtime.selectgo(0xc00029bdf0, 0xc00029bdd0, 0x138e9c8?, 0x0, 0xc0000d1ed0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00029bd58 sp=0xc00029bc38 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc0003823c0, {0x138ca08, 0xc000168140}, 0xc00038c300)
        /restic/internal/archiver/file_saver.go:255 +0x1ee fp=0xc00029bf48 sp=0xc00029bd58 pc=0x7c038e
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:54 +0x25 fp=0xc00029bf78 sp=0xc00029bf48 pc=0x7bed65
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc00029bfe0 sp=0xc00029bf78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00029bfe8 sp=0xc00029bfe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 97 [select]:
runtime.gopark(0xc000231df0?, 0x2?, 0x80?, 0xc7?, 0xc000231dd4?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000231c38 sp=0xc000231c18 pc=0x43be2e
runtime.selectgo(0xc000231df0, 0xc000231dd0, 0x138e9c8?, 0x0, 0xc000231ed0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000231d58 sp=0xc000231c38 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc0003823c0, {0x138ca08, 0xc000168140}, 0xc00038c300)
        /restic/internal/archiver/file_saver.go:255 +0x1ee fp=0xc000231f48 sp=0xc000231d58 pc=0x7c038e
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:54 +0x25 fp=0xc000231f78 sp=0xc000231f48 pc=0x7bed65
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000231fe0 sp=0xc000231f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000231fe8 sp=0xc000231fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 98 [select]:
runtime.gopark(0xc00023b610?, 0x2?, 0x0?, 0x0?, 0xc00023b584?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000118c20 sp=0xc000118c00 pc=0x43be2e
runtime.selectgo(0xc000118e10, 0xc00023b580, 0xc0001506c0?, 0x0, 0x8?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000118d40 sp=0xc000118c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x7?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000118f48 sp=0xc000118d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000118f78 sp=0xc000118f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000118fe0 sp=0xc000118f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000118fe8 sp=0xc000118fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 99 [select]:
runtime.gopark(0xc000392610?, 0x2?, 0x0?, 0x0?, 0xc000392584?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00025bc20 sp=0xc00025bc00 pc=0x43be2e
runtime.selectgo(0xc00025be10, 0xc000392580, 0x138ca08?, 0x0, 0x1386400?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00025bd40 sp=0xc00025bc20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc00025bf48 sp=0xc00025bd40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc00025bf78 sp=0xc00025bf48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc00025bfe0 sp=0xc00025bf78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00025bfe8 sp=0xc00025bfe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 100 [select]:
runtime.gopark(0xc000071e10?, 0x2?, 0x0?, 0x0?, 0xc000071d84?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000117c20 sp=0xc000117c00 pc=0x43be2e
runtime.selectgo(0xc000117e10, 0xc000071d80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000117d40 sp=0xc000117c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000117f48 sp=0xc000117d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000117f78 sp=0xc000117f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000117fe0 sp=0xc000117f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 101 [select]:
runtime.gopark(0xc0000de610?, 0x2?, 0x0?, 0x0?, 0xc0000de584?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000114c20 sp=0xc000114c00 pc=0x43be2e
runtime.selectgo(0xc000114e10, 0xc0000de580, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000114d40 sp=0xc000114c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000114f48 sp=0xc000114d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000114f78 sp=0xc000114f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000114fe0 sp=0xc000114f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000114fe8 sp=0xc000114fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 102 [select]:
runtime.gopark(0xc0000dee10?, 0x2?, 0x0?, 0x0?, 0xc0000ded84?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000115c20 sp=0xc000115c00 pc=0x43be2e
runtime.selectgo(0xc000115e10, 0xc0000ded80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000115d40 sp=0xc000115c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000115f48 sp=0xc000115d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000115f78 sp=0xc000115f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000115fe0 sp=0xc000115f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 103 [select]:
runtime.gopark(0xc0000df610?, 0x2?, 0x0?, 0x0?, 0xc0000df584?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000229c20 sp=0xc000229c00 pc=0x43be2e
runtime.selectgo(0xc000229e10, 0xc0000df580, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000229d40 sp=0xc000229c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000229f48 sp=0xc000229d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000229f78 sp=0xc000229f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000229fe0 sp=0xc000229f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000229fe8 sp=0xc000229fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 104 [select]:
runtime.gopark(0xc0000dfe10?, 0x2?, 0x0?, 0x0?, 0xc0000dfd84?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000223c20 sp=0xc000223c00 pc=0x43be2e
runtime.selectgo(0xc000223e10, 0xc0000dfd80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000223d40 sp=0xc000223c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000223f48 sp=0xc000223d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000223f78 sp=0xc000223f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000223fe0 sp=0xc000223f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000223fe8 sp=0xc000223fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 105 [select]:
runtime.gopark(0xc0000e0610?, 0x2?, 0x0?, 0x0?, 0xc0000e0584?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000222c20 sp=0xc000222c00 pc=0x43be2e
runtime.selectgo(0xc000222e10, 0xc0000e0580, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000222d40 sp=0xc000222c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000222f48 sp=0xc000222d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000222f78 sp=0xc000222f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000222fe0 sp=0xc000222f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000222fe8 sp=0xc000222fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 106 [select]:
runtime.gopark(0xc0000e0e10?, 0x2?, 0x0?, 0x0?, 0xc0000e0d84?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000224c20 sp=0xc000224c00 pc=0x43be2e
runtime.selectgo(0xc000224e10, 0xc0000e0d80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000224d40 sp=0xc000224c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000224f48 sp=0xc000224d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000224f78 sp=0xc000224f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000224fe0 sp=0xc000224f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000224fe8 sp=0xc000224fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

goroutine 107 [select]:
runtime.gopark(0xc0000e1610?, 0x2?, 0x0?, 0x0?, 0xc0000e1584?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000257c20 sp=0xc000257c00 pc=0x43be2e
runtime.selectgo(0xc000257e10, 0xc0000e1580, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000257d40 sp=0xc000257c20 pc=0x44b8e5
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x0?, {0x138ca08, 0xc000168140}, 0xc00038c360)
        /restic/internal/archiver/tree_saver.go:152 +0x1b7 fp=0xc000257f48 sp=0xc000257d40 pc=0x7c49f7
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x25 fp=0xc000257f78 sp=0xc000257f48 pc=0x7c3a85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:75 +0x56 fp=0xc000257fe0 sp=0xc000257f78 pc=0x770a36
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000257fe8 sp=0xc000257fe0 pc=0x46bea1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 87
        /home/build/go/pkg/mod/golang.org/x/sync@v0.5.0/errgroup/errgroup.go:72 +0x96

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x0
rdx    0x0
rdi    0x1b6c948
rsi    0x80
rbp    0x7ffff93296b8
rsp    0x7ffff9329670
r8     0x0
r9     0x0
r10    0x0
r11    0x0
r12    0x3
r13    0xc000255eaa
r14    0x1b6c1e0
r15    0x0
rip    0x46dcc3
rflags 0x286
cs     0x33
fs     0x53
gs     0x2b

@fd0
Copy link
Member

fd0 commented Mar 16, 2024 •
edited

Thanks a lot! I requested the stack trace on Mastodon (because I was curious), at least one Goroutine hangs in an IO wait (stack trace formatted with panicparse):

1: IO wait [Created by errgroup.(*Group).Go in goroutine 1 @ errgroup.go:72]
    runtime    proc.go:398           gopark(func(#15), Pointer(0x0), waitReason(0xc))
    runtime    netpoll.go:564        netpollblock(*pollDesc(#18), 4226534, false)
    poll       netpoll.go:343        runtime_pollWait(*pollDesc(#208), 114)
    poll       fd_poll_runtime.go:84 (*pollDesc).wait(*pollDesc(#173), 824633999456, true)
    poll       fd_poll_runtime.go:89 (*pollDesc).waitRead(...)
    poll       fd_unix.go:164        (*FD).Read(*FD(#173), []byte(#50 len=43 cap=43))
    os         file_posix.go:29      (*File).read(...)
    os         file.go:118           (*File).Read(*File(#121), []byte(#50 len=824634570560 cap=4523529))
    io         io.go:335             ReadAtLeast(Reader{#27, #121}, []byte(#50 len=43 cap=43), 43)
    io         io.go:354             ReadFull(...)
    main       exclude.go:190        isDirExcludedByFile({#196, #99}, {#24, 9}, {#25, 0x2b})
    main       exclude.go:158        isExcludedByFile({#196, #72}, {#24, 0xc}, {#25, 0x2b}, #191)
    main       exclude.go:133        rejectIfPresent.func1({#196, 0xa})
    main       cmd_backup.go:559     runBackup.func1({#196, 0xa})
    archiver   scanner.go:107        (*Scanner).scan(#198, {#2, #70}, {8, 3, 0xd2, 8}, {#196, 0xa})
    archiver   scanner.go:134        (*Scanner).scan(#198, {#2, #70}, {4, 0, 4, #194}, {#195, 8})
    archiver   scanner.go:134        (*Scanner).scan(#198, {#2, #70}, {#22, #97, #73, #20}, {#194, 4})
    archiver   scanner.go:49         (*Scanner).scanTree(#198, {#2, #70}, {#170, 0x1b, 0x9b6f2ec400000300, #120}, {0, {#194, 4}, ...})
    archiver   scanner.go:60         (*Scanner).scanTree(#198, {#2, #70}, {#152, #16, #33, #122}, {#98, {0, 0}, ...})
    archiver   scanner.go:91         (*Scanner).Scan(#198, {#2, #70}, {#6, 1, 1})
    main       cmd_backup.go:623     runBackup.func5()
    errgroup   errgroup.go:75        (*Group).Go.func1()
    runtime    asm_amd64.s:1650      goexit()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@scy @fd0