Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker file doesn't contain zstd / invalid input: magic number mismatch #13

Closed
8secz-johndpope opened this issue Oct 12, 2020 · 1 comment · Fixed by #14
Closed

Docker file doesn't contain zstd / invalid input: magic number mismatch #13

8secz-johndpope opened this issue Oct 12, 2020 · 1 comment · Fixed by #14

Comments

@8secz-johndpope
Copy link

8secz-johndpope commented Oct 12, 2020 •
edited
Loading

/data # zstd
/bin/sh: zstd: not found

it makes it hard to get the github-ranks-backup.bin on docker.

cd /data
wget https://github.com/Vivino/rankdb/releases/download/sample/github-ranks-backup.bin.zst
tar -xf github-ranks-backup.bin.zst // won't work properly
apk  add zstd # FIXES THINGS
unzstd github-ranks-backup.bin.zst

where to put the file so the api process can find file? ??? I tried to use the data / api / usr/bin
on imac we don't have /mnt
so I start using

docker run -e "GOGC=25" -p 8080:8080 -v data:/data -v conf:/conf vivino/rankdb


mv github-ranks-backup.bin /usr/bin/
rankdb-cli --timeout=5m restore multilist --src="file" --src_file="github-ranks-backup.bin"


rankdb-cli --timeout=5m restore multilist --src="file" --src_file="/usr/bin/github-ranks-backup.bin"
2020/10/12 03:48:46 [INFO] started id=xPl5tjZi POST=http://localhost:8080/xlist/restore?src=file&src_file=%2Fusr%2Fbin%2Fgithub-ranks-backup.bin
2020/10/12 03:48:46 [INFO] completed id=xPl5tjZi status=400 time=10.059511ms
error: 400: {"id":"VDCb+C3I","code":"bad_request","status":400,"detail":"invalid input: magic number mismatch"}

UPDATE - file can be found by api process - but doesn't seem golang zstd is compatible with extracted contents from zstd.

unzstd github-ranks-backup.bin.zst 
github-ranks-backup.bin.zst: 1862623243 bytes                                  
/data # ls
000000.vlog                  LOCK                         github-ranks-backup.bin      zstd-1.4.5                   zstd-bin-1.4.4-1.tar.bz2
000002.sst                   MANIFEST                     github-ranks-backup.bin.zst  zstd-1.4.5.tar.gz
/data # rankdb-cli --timeout=5m restore multilist --src="file" --src_file="/data/github-ranks-backup.bin"
2020/10/12 04:01:31 [INFO] started id=nRshlp9N POST=http://localhost:8080/xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bin
2020/10/12 04:01:31 [INFO] completed id=nRshlp9N status=400 time=8.648191ms
error: 400: {"id":"YwnsF7V2","code":"bad_request","status":400,"detail":"invalid input: magic number mismatch"}
/data # rankdb-cli --timeout=5m restore multilist --src="file" --src_file="/data/github-ranks-backup.bina"
2020/10/12 04:01:35 [INFO] started id=6Hrh1UWq POST=http://localhost:8080/xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bina
2020/10/12 04:01:35 [INFO] completed id=6Hrh1UWq status=400 time=2.842663ms
**error: 400: {"id":"aHuGXDbU","code":"bad_request","status":400,"detail":"open /data/github-ranks-backup.bina: no such file or directory"}** indicating it can find it.
/data # rankdb-cli --timeout=5m restore multilist --src="file" --src_file="/data/github-ranks-backup.bin"
2020/10/12 04:01:39 [INFO] started id=UDHi4T01 POST=http://localhost:8080/xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bin
2020/10/12 04:01:39 [INFO] completed id=UDHi4T01 status=400 time=2.373612ms
error: 400: {"id":"1KD2vX7K","code":"bad_request","status":400,"detail":"invalid input: magic number mismatch"}

related
https://github.com/klauspost/compress/blob/master/zstd/zstd.go#L49

UPDATE2.

/data # apk --no-cache add --update tar
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/1) Installing tar (1.32-r0)
Executing busybox-1.30.1-r3.trigger
OK: 245 MiB in 79 packages
/data # tar -I zstd -xvf github-ranks-backup.bin.zst 
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

seems like file didn't download correctly. my bad

/data # ls -l github-ranks-backup.bin
-rw-r--r-- 1 root root 1862623243 Oct 12 04:11 github-ranks-backup.bin

UPDATE
file is correctly extracting to bin file - but not sure why it can't read it.....
keeps throwing
error: 400: {"id":"V4wKCvKw","code":"bad_request","status":400,"detail":"invalid input: magic number mismatch"}
/data # sha1sum github-ranks-backup.bin
06e091675e72219e97a8a4e7e25e84efbc416251 github-ranks-backup.bin

UPDATE - some progress reverting to use this - some progress.
rankdb-cli --timeout=5m restore multilist --src="file" --src_file="/data/github-ranks-backup.bin.zst"

unfortunately doesn't succeed and seems to shutdown docker container

UPDATE - PROGRESS

cd /data
wget https://github.com/Vivino/rankdb/releases/download/sample/github-ranks-backup.bin.zst
rankdb-cli --timeout=5m restore multilist --src="file" --src_file="/data/github-ranks-backup.bin.zst"


BUT KEEP GETTING WARN: Lock expired!
INFO[6360] WARN: Lock expired! [POST /xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bin.zst x2djHyMvRp-130]. Called from /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/shutdown.go:27 service=shutdown




/ # rankdb-cli --timeout=5m restore multilist --src="file" --src_file="/data/github-ranks-backup.bin.zst"
2020/10/12 04:27:20 [INFO] started id=nt1A+nLl POST=http://localhost:8080/xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bin.zst
Saving session...completed.

[Process completed]



UPDATE - IT SEEMS TO HAVE FINISHED....I attempted to bump the timeout - but it instantly finished....
N.B. I increased RAM from 2GB -> 6GB. this may have made the difference

/data # rankdb-cli --timeout=500m restore multilist --src="file" --src_file="/data/github-ranks-backup.bin.zst"
2020/10/12 04:44:51 [INFO] started id=/g1TrqLq POST=http://localhost:8080/xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bin.zst
2020/10/12 04:45:24 [INFO] completed id=/g1TrqLq status=200 time=32.656310995s
{"errors":null,"restored":16,"skipped":0}



INFO[6342] started POST="/xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bin.zst" action=restore ctrl=MultilistController from=127.0.0.1 req_id=x2djHyMvRp-130

INFO[6342] Restoring list dest_list_id=repos list_id=repos

INFO[6360] Saving lists service="Manager:IntervalSaver"

goroutine profile: total 37

4 @ 0x431230 0x405bc8 0x405b9e 0x4058cb 0x8a194e 0x45f3f1

INFO[6360] Saved lists lists=0 service="Manager:IntervalSaver" size=16

# 0x8a194d github.com/Vivino/rankdb/blobstore.(*LazySaver).startSaver+0x2bd /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/lazysaver.go:465


3 @ 0x431230 0x405bc8 0x405b9e 0x4058cb 0x8be649 0x45f3f1

# 0x8be648 github.com/klauspost/compress/zstd.(*blockDec).startDecoder+0x128 /go/pkg/mod/github.com/klauspost/compress@v1.9.1/zstd/blockdec.go:190


3 @ 0x431230 0x440e6b 0xaad24e 0x45f3f1

# 0xaad24d github.com/klauspost/shutdown2.onFunc.func1+0xcd /go/pkg/mod/github.com/klauspost/shutdown2@v1.1.0/shutdown.go:318


3 @ 0x431230 0x441dd0 0x441dbb 0x41b43e 0x41b422 0x41f095 0x40c9a7 0x40cc28 0x89e828 0x89e7e2 0x8a6242 0x8a54b0 0x8a83f7 0x8a1ab7 0x45f3f1

# 0x89e827 math/rand.NewSource+0x67 /go/pkg/mod/github.com/cenkalti/backoff@v0.0.0-20170921171023-61ba96c4d100/exponential.go:92

# 0x89e7e1 github.com/cenkalti/backoff.NewExponentialBackOff+0x21 /go/pkg/mod/github.com/cenkalti/backoff@v0.0.0-20170921171023-61ba96c4d100/exponential.go:92

# 0x8a6241 github.com/Vivino/rankdb/blobstore.glob..func12+0x21 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/retry.go:34

# 0x8a54af github.com/Vivino/rankdb/blobstore.(*RetryStore).Set+0x3f /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/retry.go:145

# 0x8a83f6 github.com/Vivino/rankdb/blobstore.(*LazySaver).startSaver.func1+0x2b6 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/lazysaver.go:482

# 0x8a1ab6 github.com/Vivino/rankdb/blobstore.(*LazySaver).startSaver+0x426 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/lazysaver.go:503


2 @ 0x431230 0x405bc8 0x405b9e 0x4058cb 0xaad37a 0x45f3f1

# 0xaad379 github.com/klauspost/shutdown2.OnSignal.func1+0x59 /go/pkg/mod/github.com/klauspost/shutdown2@v1.1.0/shutdown.go:376


2 @ 0x431230 0x440e6b 0xd301a6 0x45f3f1

# 0xd301a5 github.com/dgraph-io/badger/y.(*WaterMark).process+0x2e5 /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/y/watermark.go:207


2 @ 0x431230 0x440e6b 0xd54dc8 0x45f3f1

# 0xd54dc7 github.com/dgraph-io/badger.(*levelsController).runWorker+0x227 /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/levels.go:348


1 @ 0x40b194 0x445e0c 0x944bd2 0x45f3f1

# 0x445e0b os/signal.signal_recv+0x9b /usr/local/go/src/runtime/sigqueue.go:147

# 0x944bd1 os/signal.loop+0x21 /usr/local/go/src/os/signal/signal_unix.go:23


1 @ 0x413649 0xd8009c 0x8a8f64 0x89efbf 0x8a5650 0x8a565b 0x8a83f7 0x8a1ab7 0x45f3f1

# 0xd8009b github.com/Vivino/rankdb/blobstore/badgerstore.(*BadgerStore).Set+0x18b /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/badgerstore/badger.go:115

# 0x8a8f63 github.com/Vivino/rankdb/blobstore.(*RetryStore).Set.func1+0xc3 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/retry.go:154

# 0x89efbe github.com/cenkalti/backoff.RetryNotify+0x9e /go/pkg/mod/github.com/cenkalti/backoff@v0.0.0-20170921171023-61ba96c4d100/retry.go:37

# 0x8a564f github.com/cenkalti/backoff.Retry+0x1df /go/pkg/mod/github.com/cenkalti/backoff@v0.0.0-20170921171023-61ba96c4d100/retry.go:25

# 0x8a565a github.com/Vivino/rankdb/blobstore.(*RetryStore).Set+0x1ea /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/retry.go:160

# 0x8a83f6 github.com/Vivino/rankdb/blobstore.(*LazySaver).startSaver.func1+0x2b6 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/lazysaver.go:482

# 0x8a1ab6 github.com/Vivino/rankdb/blobstore.(*LazySaver).startSaver+0x426 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/lazysaver.go:503


1 @ 0x431230 0x405bc8 0x405b9e 0x40588b 0x8c84d2 0x45f3f1

# 0x8c84d1 github.com/klauspost/compress/zstd.(*Decoder).startStreamDecoder+0x181 /go/pkg/mod/github.com/klauspost/compress@v1.9.1/zstd/decoder.go:452


1 @ 0x431230 0x405bc8 0x405b9e 0x40588b 0x8cec1b 0x45f3f1

# 0x8cec1a github.com/klauspost/compress/zstd.(*frameDec).startDecoder+0x27a /go/pkg/mod/github.com/klauspost/compress@v1.9.1/zstd/framedec.go:432


1 @ 0x431230 0x405bc8 0x405b9e 0x40588b 0xd72080 0x45f3f1

# 0xd7207f github.com/dgraph-io/badger.(*valueLog).waitOnGC+0x6f /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/value.go:1337


1 @ 0x431230 0x405bc8 0x405b9e 0x40588b 0xdb5e03 0x45f3f1

# 0xdb5e02 github.com/Vivino/rankdb/api.(*HealthController).memoryReader+0xb2 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/health.go:68


1 @ 0x431230 0x405bc8 0x405b9e 0x4058cb 0xd48043 0xd74997 0x45f3f1

# 0xd48042 github.com/dgraph-io/badger.(*DB).flushMemtable+0xc2 /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/db.go:912

# 0xd74996 github.com/dgraph-io/badger.Open.func4+0x36 /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/db.go:298


1 @ 0x431230 0x42bffa 0x42b5c5 0x4da755 0x4db6cf 0x4db6b1 0x5d5e0f 0x5e9318 0x6f0fe8 0x45f3f1

# 0x42b5c4 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184

# 0x4da754 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87

# 0x4db6ce internal/poll.(*pollDesc).waitRead+0x1ce /usr/local/go/src/internal/poll/fd_poll_runtime.go:92

# 0x4db6b0 internal/poll.(*FD).Read+0x1b0 /usr/local/go/src/internal/poll/fd_unix.go:169

# 0x5d5e0e net.(*netFD).Read+0x4e /usr/local/go/src/net/fd_unix.go:202

# 0x5e9317 net.(*conn).Read+0x67 /usr/local/go/src/net/net.go:184

# 0x6f0fe7 net/http.(*connReader).backgroundRead+0x57 /usr/local/go/src/net/http/server.go:688


1 @ 0x431230 0x42bffa 0x42b5c5 0x4da755 0x4dd0d8 0x4dd0b7 0x5d6782 0x5f3782 0x5f2307 0x6fb9d0 0xdc47c4 0xdce1bf 0x430e76 0x45f3f1

# 0x42b5c4 internal/poll.runtime_pollWait+0x54 /usr/local/go/src/runtime/netpoll.go:184

# 0x4da754 internal/poll.(*pollDesc).wait+0x44 /usr/local/go/src/internal/poll/fd_poll_runtime.go:87

# 0x4dd0d7 internal/poll.(*pollDesc).waitRead+0x1f7 /usr/local/go/src/internal/poll/fd_poll_runtime.go:92

# 0x4dd0b6 internal/poll.(*FD).Accept+0x1d6 /usr/local/go/src/internal/poll/fd_unix.go:384

# 0x5d6781 net.(*netFD).accept+0x41 /usr/local/go/src/net/fd_unix.go:238

# 0x5f3781 net.(*TCPListener).accept+0x31 /usr/local/go/src/net/tcpsock_posix.go:139

# 0x5f2306 net.(*TCPListener).Accept+0x46 /usr/local/go/src/net/tcpsock.go:261

# 0x6fb9cf net/http.(*Server).Serve+0x27f /usr/local/go/src/net/http/server.go:2925

# 0xdc47c3 github.com/Vivino/rankdb/api.StartServices+0xa13 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/services.go:98

# 0xdce1be main.main+0x50e /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/cmd/rankdb/main.go:77

# 0x430e75 runtime.main+0x205 /usr/local/go/src/runtime/proc.go:203


1 @ 0x431230 0x440e6b 0x8a0f64 0x45f3f1

# 0x8a0f63 github.com/Vivino/rankdb/blobstore.(*LazySaver).requestHandler+0x733 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/blobstore/lazysaver.go:418


1 @ 0x431230 0x440e6b 0x916f0f 0x45f3f1

# 0x916f0e github.com/Vivino/rankdb.(*Manager).StartListPruner.func2+0x1de /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/manager.go:114


1 @ 0x431230 0x440e6b 0x9173ad 0x45f3f1

# 0x9173ac github.com/Vivino/rankdb.(*Manager).StartListSplitter.func1+0x1fc /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/manager.go:140


1 @ 0x431230 0x440e6b 0xd469b8 0x45f3f1

# 0xd469b7 github.com/dgraph-io/badger.(*DB).doWrites+0x417 /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/db.go:705


1 @ 0x431230 0x440e6b 0xd6023f 0x45f3f1

# 0xd6023e github.com/dgraph-io/badger.(*publisher).listenForUpdates+0x17e /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/publisher.go:65


1 @ 0x431230 0x441dd0 0x441dbb 0x41b43e 0x41b422 0x41f095 0x40c9a7 0x40cc28 0x8dd9e6 0x907f50 0x8ea72c 0x8ea453 0x8f540d 0xaaf65f 0xdbf657 0xa6c29e 0xdc5747 0xa6cd15 0x9a3079 0xdb029f 0xdc91cd 0xdaeb42 0xdabe66 0xdc7dee 0xdad829 0xdaed40 0xdc8c11 0x9a3584 0x9a2175 0x994619 0x994739 0x99f31c

# 0x8dd9e5 github.com/Vivino/rankdb.(*Elements).Sort+0x3a5 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/elements.go:140

# 0x907f4f github.com/Vivino/rankdb.(*Segments).ElementIndexAll+0x41f /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/segments.go:832

# 0x8ea72b github.com/Vivino/rankdb.(*List).reindex+0x25b /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/list.go:748

# 0x8ea452 github.com/Vivino/rankdb.(*List).Reindex+0x152 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/list.go:729

# 0x8f540c github.com/Vivino/rankdb.RestoreList+0x73c /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/list.go:2005

# 0xaaf65e github.com/Vivino/rankdb/backup.(*RestoreInfo).Restore+0x60e /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/backup/restore.go:97

# 0xdbf656 github.com/Vivino/rankdb/api.(*MultilistController).Restore+0x236 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/multilist.go:457

# 0xa6c29d github.com/Vivino/rankdb/api/app.MountMultilistController.func7+0xfd /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/app/controllers.go:778

# 0xdc5746 github.com/Vivino/rankdb/api.NewJWTMiddleware.func1.1+0x56 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/jwt.go:45

# 0xa6cd14 github.com/Vivino/rankdb/api/app.handleSecurity.func1+0x174 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/app/security.go:55

# 0x9a3078 github.com/goadesign/goa.(*Controller).MuxHandler.func1.1.1+0xc8 /go/pkg/mod/github.com/goadesign/goa@v1.4.1/service.go:309

# 0xdb029e github.com/goadesign/goa/middleware/gzip.Middleware.func2.1+0x29e /go/pkg/mod/github.com/goadesign/goa@v1.4.1/middleware/gzip/middleware.go:308

# 0xdc91cc github.com/Vivino/rankdb/api.ShutdownMiddleware.func1+0x23c /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/shutdown.go:32

# 0xdaeb41 github.com/goadesign/goa/middleware.Recover.func1.1+0xb1 /go/pkg/mod/github.com/goadesign/goa@v1.4.1/middleware/recover.go:37

# 0xdabe65 github.com/goadesign/goa/middleware.ErrorHandler.func1.1+0x95 /go/pkg/mod/github.com/goadesign/goa@v1.4.1/middleware/error_handler.go:21

# 0xdc7ded github.com/Vivino/rankdb/api.NewRelicTx.func1.1+0x78d /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/newrelic.go:139

# 0xdad828 github.com/goadesign/goa/middleware.LogRequest.func1.1+0xdd8 /go/pkg/mod/github.com/goadesign/goa@v1.4.1/middleware/log_request.go:96

# 0xdaed3f github.com/goadesign/goa/middleware.RequestIDWithHeaderAndLengthLimit.func1.1+0x12f /go/pkg/mod/github.com/goadesign/goa@v1.4.1/middleware/request_id.go:63

# 0xdc8c10 github.com/Vivino/rankdb/api.SetLogger.func1.1+0x2e0 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/rankdb.go:355

# 0x9a3583 github.com/goadesign/goa.(*Controller).MuxHandler.func1+0x243 /go/pkg/mod/github.com/goadesign/goa@v1.4.1/service.go:343

# 0x9a2174 github.com/goadesign/goa.(*mux).Handle.func1+0x1c4 /go/pkg/mod/github.com/goadesign/goa@v1.4.1/mux.go:69

# 0x994618 github.com/dimfeld/httptreemux.(*TreeMux).ServeLookupResult+0x168 /go/pkg/mod/github.com/dimfeld/httptreemux@v4.0.1+incompatible/router.go:245

# 0x994738 github.com/dimfeld/httptreemux.(*TreeMux).ServeHTTP+0x108 /go/pkg/mod/github.com/dimfeld/httptreemux@v4.0.1+incompatible/router.go:266

# 0x99f31b github.com/goadesign/goa.(*mux).ServeHTTP+0x4b /go/pkg/mod/github.com/goadesign/goa@v1.4.1/mux.go:100


1 @ 0x4460a6 0x8cbc62 0x8cb6e4 0x8fccad 0x8ff5a3 0x916922 0x45f3f1

# 0x8cbc61 github.com/klauspost/compress/zstd.(*Encoder).Reset+0x3b1 /go/pkg/mod/github.com/klauspost/compress@v1.9.1/zstd/encoder.go:101

# 0x8cb6e3 github.com/klauspost/compress/zstd.NewWriter+0x1b3 /go/pkg/mod/github.com/klauspost/compress@v1.9.1/zstd/encoder.go:72

# 0x8fccac github.com/Vivino/rankdb.Lists.Save+0x18c /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/lists.go:101

# 0x8ff5a2 github.com/Vivino/rankdb.(*Manager).SaveLists+0x92 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/manager.go:188

# 0x916921 github.com/Vivino/rankdb.(*Manager).StartIntervalSaver.func1+0x351 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/manager.go:78


1 @ 0x48d1f5 0x48c396 0x4e81ce 0x4e81b8 0x4e7b6d 0x522ed3 0x52325f 0xd751c2 0xd48462 0xd4869e 0x45f3f1

# 0x48d1f4 syscall.Syscall6+0x4 /usr/local/go/src/syscall/asm_linux_amd64.s:44

# 0x48c395 syscall.fstatat+0xc5 /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1520

# 0x4e81cd syscall.Lstat+0x6d /usr/local/go/src/syscall/syscall_linux_amd64.go:72

# 0x4e81b7 os.lstatNolog+0x57 /usr/local/go/src/os/stat_unix.go:42

# 0x4e7b6c os.Lstat+0x4c /usr/local/go/src/os/stat.go:22

# 0x522ed2 path/filepath.walk+0x1d2 /usr/local/go/src/path/filepath/path.go:376

# 0x52325e path/filepath.Walk+0xfe /usr/local/go/src/path/filepath/path.go:404

# 0xd751c1 github.com/dgraph-io/badger.(*DB).calculateSize.func2+0x91 /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/db.go:964

# 0xd48461 github.com/dgraph-io/badger.(*DB).calculateSize+0x61 /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/db.go:982

# 0xd4869d github.com/dgraph-io/badger.(*DB).updateSize+0xcd /go/pkg/mod/github.com/dgraph-io/badger@v1.6.0/db.go:1000


1 @ 0x52a295 0x52a0b0 0x526cda 0xdce54c 0x46dce3 0xdce5c1 0xdce586 0xaad671 0x45f3f1

# 0x52a294 runtime/pprof.writeRuntimeProfile+0x94 /usr/local/go/src/runtime/pprof/pprof.go:708

# 0x52a0af runtime/pprof.writeGoroutine+0x9f /usr/local/go/src/runtime/pprof/pprof.go:670

# 0x526cd9 runtime/pprof.(*Profile).WriteTo+0x3d9 /usr/local/go/src/runtime/pprof/pprof.go:329

# 0xdce54b main.main.func1.1+0x6b /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/cmd/rankdb/main.go:57

# 0x46dce2 sync.(*Once).doSlow+0xe2 /usr/local/go/src/sync/once.go:66

# 0xdce5c0 sync.(*Once).Do+0x60 /usr/local/go/src/sync/once.go:57

# 0xdce585 main.main.func1+0x25 /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/cmd/rankdb/main.go:56

# 0xaad670 github.com/klauspost/shutdown2.Lock.func1+0x290 /go/pkg/mod/github.com/klauspost/shutdown2@v1.1.0/shutdown.go:557


INFO[6360] Saving done duration=53.731755ms service="Manager:IntervalSaver"

INFO[6360] WARN: Lock expired! [POST /xlist/restore?src=file&src_file=%2Fdata%2Fgithub-ranks-backup.bin.zst x2djHyMvRp-130]. Called from /go/pkg/mod/github.com/!vivino/rankdb@v1.0.4/api/shutdown.go:27 service=shutdown
@8secz-johndpope 8secz-johndpope changed the title Docker file doesn't contain zstd / invalid input: magic number mismatch / can Docker file doesn't contain zstd / invalid input: magic number mismatch Oct 12, 2020
klauspost added a commit to klauspost/rankdb that referenced this issue Oct 12, 2020
@klauspost
Update zstd
c4d53c0 
Fixes Vivino#13

Includes fix from klauspost/compress#278
@klauspost klauspost mentioned this issue Oct 12, 2020
Merged
eikmadsen pushed a commit that referenced this issue Oct 20, 2020
@klauspost
Update zstd fix ReadFrom with small blocks
00609f2 
Fixes #13

Includes fix from klauspost/compress#278
@eikmadsen
Copy link
Contributor

@8secz-johndpope I merged a PR from @klauspost that should fix the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update zstd klauspost/rankdb
2 participants
@eikmadsen @8secz-johndpope