Unable to update from 2019-08-14 to 2019-08-21 #8174

mymasse · 2019-09-03T16:30:07Z

Using the MinIO Bosh releases we are unable to upgrade from version 2019-08-14 to 2019-08-21. We are using a deployment equivalent to the sample manifest provided in the bosh release repositoty: 4 VMs with 1 disk each.

Expected Behavior

Cluster should start successfully

Current Behavior

Cluster doesn't initialize and reports the following in the logs

Your Environment

Version used: RELEASE.2019-08-14T20-37-41Z
Environment name and version (e.g. nginx 1.9.1): Bosh with Ubuntu Xenial Stemcell (456.12)
Server type and version: Bosh with Ubuntu Xenial Stemcell (456.12) on vSphere IAAS
Operating System and version (uname -a): Linux 8f6fb5cc-3b97-459a-a7f1-3a2386f79eda 4.15.0-55-generic Adding more api suite tests #60~16.04.2-Ubuntu SMP Thu Jul 4 09:03:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Link to your project: None

Error logs

API: SYSTEM()
Time: 17:20:51 UTC 08/26/2019
DeploymentID: b7655837-69e3-4d97-bf1f-2935d364b468
Error: Bitrot verification mismatch - expected 6e1488613d8b2bd0df52f853f32a7a96f7943e8fee14b0b84a89deb4732b8156, received 61530498dec22770675f15a140e44634eb28787040863ad84307f89f43e8f950
       2: cmd/bitrot-whole.go:75:cmd.(*wholeBitrotReader).ReadAt()
       1: cmd/erasure-decode.go:110:cmd.(*parallelReader).Read.func1()

API: SYSTEM()
Time: 17:20:51 UTC 08/26/2019
DeploymentID: b7655837-69e3-4d97-bf1f-2935d364b468
Error: Bitrot verification mismatch - expected 6e1488613d8b2bd0df52f853f32a7a96f7943e8fee14b0b84a89deb4732b8156, received 221f0c03e192ea70a7b2c7a2e4cddcedbcf7533eb8814c342f970e8e003b26a4
       2: cmd/bitrot-whole.go:75:cmd.(*wholeBitrotReader).ReadAt()
       1: cmd/erasure-decode.go:110:cmd.(*parallelReader).Read.func1()

API: SYSTEM()
Time: 17:20:51 UTC 08/26/2019
DeploymentID: b7655837-69e3-4d97-bf1f-2935d364b468
Error: Bitrot verification mismatch - expected 6e1488613d8b2bd0df52f853f32a7a96f7943e8fee14b0b84a89deb4732b8156, received e704b2481dc84c9e77414c7c7da187868fa7556735417d19f751d7e02bfb38bb
       2: cmd/bitrot-whole.go:75:cmd.(*wholeBitrotReader).ReadAt()
       1: cmd/erasure-decode.go:110:cmd.(*parallelReader).Read.func1()

API: SYSTEM()
Time: 17:20:51 UTC 08/26/2019
DeploymentID: b7655837-69e3-4d97-bf1f-2935d364b468
Error: Storage resources are insufficient for the read operation.
       7: cmd/config-common.go:40:cmd.readConfig()
       6: cmd/iam-object-store.go:223:cmd.(*IAMObjectStore).loadIAMConfig()
       5: cmd/iam-object-store.go:245:cmd.(*IAMObjectStore).loadPolicyDoc()
       4: cmd/iam-object-store.go:267:cmd.(*IAMObjectStore).loadPolicyDocs()
       3: cmd/iam-object-store.go:432:cmd.(*IAMObjectStore).loadAll()
       2: cmd/iam.go:375:cmd.(*IAMSys).Init()
       1: cmd/server-main.go:349:cmd.serverMain()
Waiting for IAM subsystem to be initialized..

The text was updated successfully, but these errors were encountered:

harshavardhana · 2019-09-03T16:31:59Z

You can't do rolling upgrades from 14 to 21 as there is breakage of internode APIs all servers have to be updated together

mymasse · 2019-09-03T16:35:10Z

OK, well bosh updates all the servers one by one and after all 4 are updated none of them work. What should be the correct steps when doing an upgrade with Bosh?

the-glu · 2019-09-05T13:18:28Z

I did get the same issues upgrading two groups of servers from 2019-07-05T21-20-21Z to 2019-08-29T00-25-01Z.

Services where stopped during the upgrade.

One has some files with bitrot verification mismatch, one doesn't start at all (since he has issues on the configuration file, .minio.sys/config/config.json

I noticed the issue using tcpdump, trying to debugg why my server wasen't starting at all.

What is strange, is that it seems to use the wrong hash when trying to get the server. Exemple:

On server3: hash==de5a64e690faa2d7f4afe22c6c5f2817d4987c81a29f1af7df66e94913a98c18
On server 5: hash==5b6709ec5acd568ef965fbe8d353f3af7c0e6c2fbd43a00594fe35cb23d7bd55
On server13: hash==8fab13bf24c6f0d1aadd281980e3f555d435dfe20794f2c4bf8df58ddcc5b75c

Server13 ask server3 with readall, he get a reply with hash==de5a64e690faa2d7f4afe22c6c5f2817d4987c81a29f1af7df66e94913a98c18 (witch is fine)

Then he ask again with readfile and the hash in url, but put the hash of server 5 in the query (and get a bitrot verification mismatch)

Here is the "raw" tcp requests with secrets removed:

POST /minio/storage/v8/opt/minio/data/0/readall?file-path=config%2Fconfig.json%2Fxl.json&instance-id=685ae97b&volume=.minio.sys HTTP/1.1
User-Agent: Go-http-client/1.1
Content-Length: 0
X-Minio-Time: 2019-09-05T12:23:52Z

HTTP/1.1 200 OK
Content-Length: 601
Content-Security-Policy: block-all-mixed-content
Vary: Origin
X-Amz-Request-Id: 
X-Xss-Protection: 1; mode=block
Date: Thu, 05 Sep 2019 12:23:52 GMT
Content-Type: text/plain; charset=utf-8

{"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":8,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"de5a64e690faa2d7f4afe22c6c5f2817d4987c81a29f1af7df66e94913a98c18"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}


POST /minio/storage/v8/opt/minio/data/0/readfile?bitrot-algo=highwayhash256&bitrot-hash=5b6709ec5acd568ef965fbe8d353f3af7c0e6c2fbd43a00594fe35cb23d7bd55&file-path=config%2Fconfig.json%2Fpart.1&instance-id=685ae97b&length=504&offset=0&volume=.minio.sys HTTP/1.1
User-Agent: Go-http-client/1.1
Content-Length: 0
X-Minio-Time: 2019-09-05T12:23:52Z

HTTP/1.1 403 Forbidden
Content-Security-Policy: block-all-mixed-content
Vary: Origin
X-Amz-Request-Id: 
X-Xss-Protection: 1; mode=block
Date: Thu, 05 Sep 2019 12:23:52 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

b3
Bitrot verification mismatch - expected 5b6709ec5acd568ef965fbe8d353f3af7c0e6c2fbd43a00594fe35cb23d7bd55, received de5a64e690faa2d7f4afe22c6c5f2817d4987c81a29f1af7df66e94913a98c18
0

Here are the content of the xl.json for my config.json file for each server:

minio2: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":9,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"64a4a561230a343fc54b29660f8b815704feca2c2aea598af5e84d992a0c88d2"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio4: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":7,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"a021b3a92cd821fd43996835a461cfcd36c9daef293afe88e12dfc23db2d302a"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio6: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":5,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"5b6709ec5acd568ef965fbe8d353f3af7c0e6c2fbd43a00594fe35cb23d7bd55"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio5: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":6,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"5f98c4e339885a2bf25a9d326caa96cbc2201e1d54909a4cbaee222533103899"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio11: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":12,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"51fc069b2fbbee121bdaf867ff144f06d8ed14f43774a3f4a3fa0430048b5b15"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio3: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":8,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"de5a64e690faa2d7f4afe22c6c5f2817d4987c81a29f1af7df66e94913a98c18"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio9: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":2,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"8c7e7c876b3ccdb7f96658bdc1903e320494b3d841a90fe884ff5b7472133dd3"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio10: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":1,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"634752ad9006eb4ca44ebd7aaf0787278a1fafa2d8abd6e25264826405fcfeb0"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio13: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":11,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"8fab13bf24c6f0d1aadd281980e3f555d435dfe20794f2c4bf8df58ddcc5b75c"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio7: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":4,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"6ec3e13df48d4de06cecacd283d4a13a07e989681da023ec33aca661843cc0c2"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio8: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":3,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"d10707d8d8dda053e293ac5efa23a60b5c62323c844cdc7d60d70d3b4e0bb570"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

minio17: {"version":"1.0.1","format":"xl","stat":{"size":3022,"modTime":"2019-02-21T20:19:34.929272737Z"},"erasure":{"algorithm":"klauspost/reedsolomon/vandermonde","data":6,"parity":6,"blockSize":10485760,"index":10,"distribution":[11,12,1,2,3,4,5,6,7,8,9,10],"checksum":[{"name":"part.1","algorithm":"highwayhash256","hash":"43977233dfc1312627390c92421ff0458e84f7f2ec2190ae942eb2e20c4e1d70"}]},"minio":{"release":"RELEASE.2018-12-13T02-04-19Z"},"meta":{"content-type":"application/json","etag":"1f3c8ac18fd0626494bc18df516d25e6"},"parts":[{"number":1,"name":"part.1","etag":"","size":3022,"actualSize":3022}]}

harshavardhana · 2019-09-05T17:51:03Z

This is an issue is not related to bit-rot, it is related to how it is reading older objects - this issue is reproducible with the following script.

#!/bin/bash                                                                                                                    
                                                                                                                               
export MINIO_ACCESS_KEY="minio"                                                                                                
export MINIO_SECRET_KEY="minio123"                                                                                                                    
for i in {01..12}; do                                                                                                          
    minio.RELEASE.2019-01-16T21-44-08Z server --address ":90${i}" http://127.0.0.1:9001/tmp/1 http://127.0.0.1:9002/tmp/2 http\
://127.0.0.1:9003/tmp/3 http://127.0.0.1:9004/tmp/4 http://127.0.0.1:9005/tmp/5 http://127.0.0.1:9006/tmp/6 http://127.0.0.1:9\
007/tmp/7 http://127.0.0.1:9008/tmp/8 http://127.0.0.1:9009/tmp/9 http://127.0.0.1:9010/tmp/10 http://127.0.0.1:9011/tmp/11 ht\
tp://127.0.0.1:9012/tmp/12 &                                                                                                   
    #minio server --address ":90${i}" http://127.0.0.1:9001/tmp/1 http://127.0.0.1:9002/tmp/2 http://127.0.0.1:9003/tmp/3 http\
://127.0.0.1:9004/tmp/4 http://127.0.0.1:9005/tmp/5 http://127.0.0.1:9006/tmp/6 http://127.0.0.1:9007/tmp/7 http://127.0.0.1:9\
008/tmp/8 http://127.0.0.1:9009/tmp/9 http://127.0.0.1:9010/tmp/10 http://127.0.0.1:9011/tmp/11 http://127.0.0.1:9012/tmp/12 & 
done

harshavardhana · 2019-09-05T20:27:26Z

#8190 - fixed here.

harshavardhana · 2019-09-06T00:05:56Z

Fixed and released..

mymasse added community triage labels Sep 3, 2019

harshavardhana self-assigned this Sep 5, 2019

harshavardhana added priority: high and removed triage labels Sep 5, 2019

harshavardhana added this to the Next Release milestone Sep 5, 2019

harshavardhana added the fixed label Sep 6, 2019

harshavardhana closed this as completed Sep 6, 2019

github-actions bot locked as resolved and limited conversation to collaborators Jan 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to update from 2019-08-14 to 2019-08-21 #8174

Unable to update from 2019-08-14 to 2019-08-21 #8174

mymasse commented Sep 3, 2019

harshavardhana commented Sep 3, 2019

mymasse commented Sep 3, 2019

the-glu commented Sep 5, 2019

harshavardhana commented Sep 5, 2019

harshavardhana commented Sep 5, 2019

harshavardhana commented Sep 6, 2019

Unable to update from 2019-08-14 to 2019-08-21 #8174

Unable to update from 2019-08-14 to 2019-08-21 #8174

Comments

mymasse commented Sep 3, 2019

Expected Behavior

Current Behavior

Your Environment

Error logs

harshavardhana commented Sep 3, 2019

mymasse commented Sep 3, 2019

the-glu commented Sep 5, 2019

harshavardhana commented Sep 5, 2019

harshavardhana commented Sep 5, 2019

harshavardhana commented Sep 6, 2019