RELEASE.2022-10-29T06-21-33Z migration from older release

[martadinata666]

RELEASE.2022-10-29T06-21-33Z got this note.
Removes remaining gateway implementations of NAS/S3 and legacy FS mode completely - refer #15929. Existing users should continue to use their current releases if they wish to continue using gateway mode.

After upgrade results
ERROR Unable to use the drive /data: Drive /data: found backend type fs, expected xl or xl-single: Invalid arguments specified

Is there migrating guide i can follow? thanks

[harshavardhana]

Don't upgrade as mentioned in the release notes. There is nothing new that you will get by upgrading.

[martadinata666]

as my minio installation is pretty small, I deploy a new one, then mc mirror from from old deployment to new one. that's the only way I can think of and the path I'm using. maybe another way? im not really sure.

[jtackaberry]

Don't upgrade as mentioned in the release notes. There is nothing new that you will get by upgrading.

Does this imply there will be a migration path introduced in the future, once there is something new that we will get by upgrading to some future version?

I assumed not, given that the fs backend code has been entirely ripped out, so like the OP I was confused and ultimately surprised that I could find no documentation on how to migrate, or even what the recommended strategy is.

@martadinata666 's suggestion to just stand up a parallel deployment and mirror the files is at least a strategy, albeit problematic when you don't have sufficient storage to float a duplicate deployment. But if that's the only viable path in order to continue to receive updates (and more importantly security fixes) for users currently on the fs backend, then IMO it should be included in the documentation.

[harshavardhana]

While there is no reason to update this time, it is not viable to stay on older version for too long. Is there a way to migrate setup using FS backend into a new one? Maybe we should open an issue and ask for documentation update regarding such migration. I could see that the question was asked in #14331 (comment), but couldn't find an answer to it, unfortunately.

Feel free to open an issue at minio/docs

[jtackaberry]

Thanks. Filed minio/docs#624

[markovendelin]

As far as I understand, mc mirror will just sync the data. I presume we would have to go bucket-by-bucket in this case. On a top of that, somehow users have to be migrated as well as all the policies. @jtackaberry - thank you for opening an issue!

Link: minio/docs#624

[chrisjenx]

Your answer is don't upgrade... you're joking right?

[harshavardhana]

Your answer is don't upgrade... you're joking right?

No need to be sarcastic @chrisjenx - you shouldn't upgrade if you don't need to. There is nothing you will get by upgrading - nothing has changed on the legacy front in 2+yrs.

If you want to build server with latest go in some manner for the release you use just install via go install -v github.com/minio/minio@<release-tag>

[chrisjenx]

Mmmm, thankfully picking a downgraded docker image fixed it.

The annoyance came from the unintuitive error message thrown. Better DevX would be to link to the upgrade doc or change log with breaking changes in it so people have a good stepping off point.

[LucasFrecia]

When running locally on a mac, updated minio through brew and had the error:

2022/11/07 08:32:28 ERROR Unable to use the drive /usr/local/var/minio: Drive /usr/local/var/minio: found backend type fs, expected xl or xl-single: Invalid arguments specified

WARNING: This "fix", is with minio data loss and was done to have my local env running faster. Read @harshavardhana comment for proper migration steps.

The error was solved by removing old minio files and reinstalling:
rm -rf /usr/local/etc/minio /usr/local/var/minio

[harshavardhana]

Please don't do things like rm -rf with your backend - please follow the migration guide https://min.io/docs/minio/linux/operations/install-deploy-manage/migrate-fs-gateway.html

[goodale]

what do we do if the mc admin update already ran and we're now on 20221108 and minio won't start? The migration steps presume the prior version is still available and running in order to do the migration. So the migration documentation isn't helpful if the binary has already been updated...

[goodale]

I was able to downgrade so I can get back to work and revisit the lengthy migration procedure at some point in the future. Here are the steps I took in case it is helpful to anyone else:

_cd to the directory where minio is currently installed_
wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio_20221024183507.0.0_amd64.deb -O minio.deb
sudo dpkg -i minio.deb
_start minio as normal_

While the move away from the gateway was announced awhile ago there is no mention of it in the actual mc admin update process that broke things as a result to the update after the 10/24 version. It would have been nice had the update script warned users of the breaking changes given it temporarily bricks things for anyone using minio gateway or fs. (The statement [Upgrades Are Non-Disruptive](https://min.io/docs/minio/linux/operations/install-deploy-manage/upgrade-minio-deployment.html?ref=docs-redirect#id2) clearly wasn't the case for me ;)

[mirekphd]

I don't think it's the only breaking change that happened since the 2022-10-29 release (or at least since our last working version container (mirekphd/ml-minio22:20221021, which is based on the upstream minio/minio container).

There is apparently one more breaking change there: write-only-once protection is put on everything, including MinIO's own config folder.

More info:
If you start the server in a "fresh" folder / PVC (i.e. one without ~/.minio.sys - you can replicate by for instance renaming it), minio fails to list any existing files (managing to list only folders as buckets), and the newly created ~/.minio.sys is apparently put in the write-once-read-many (WORM) mode... this is how minio/minio is now configured.

Expand detailed error log

WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
WARNING: Detected Linux kernel version older than 4.0.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x.x linux kernel version for best performance
Formatting 1st pool, 1 set(s), 1 drives per set.
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
Warning: Default parity set to 0. This can lead to data loss.

API: SYSTEM()
Time: 08:02:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/config/config.json) returned an error (file access denied) (*fmt.wrapError)
      17: internal/logger/logger.go:258:logger.LogIf()
      16: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      15: internal/logger/logonce.go:135:logger.LogOnceIf()
      14: cmd/erasure-object.go:555:cmd.readAllXL()
      13: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
      12: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
      11: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
      10: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       9: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       8: cmd/config-common.go:59:cmd.readConfig()
       7: cmd/config-migrate.go:2506:cmd.checkConfigVersion()
       6: cmd/config-migrate.go:2479:cmd.migrateMinioSysConfig()
       5: cmd/config.go:225:cmd.initConfig()
       4: cmd/config.go:195:cmd.(*ConfigSys).Init()
       3: cmd/server-main.go:444:cmd.initConfigSubsystem()
       2: cmd/server-main.go:411:cmd.initServer()
       1: cmd/server-main.go:597:cmd.serverMain()

API: SYSTEM()
Time: 08:02:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Unable to initialize config, some features may be missing migrateMinioSysConfig: Prefix access is denied: .minio.sys/config/config.json (*fmt.wrapError)
       4: internal/logger/logger.go:258:logger.LogIf()
       3: cmd/server-main.go:450:cmd.initConfigSubsystem()
       2: cmd/server-main.go:411:cmd.initServer()
       1: cmd/server-main.go:597:cmd.serverMain()

API: SYSTEM()
Time: 08:02:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: open /data/.minio.sys/config/iam/format.json/xl.meta: operation not permitted (*fs.PathError)
       5: internal/logger/logger.go:258:logger.LogIf()
       4: cmd/metacache-walk.go:327:cmd.(*xlStorage).WalkDir.func2()
       3: cmd/metacache-walk.go:351:cmd.(*xlStorage).WalkDir()
       2: cmd/metacache-walk.go:361:cmd.(*xlStorageDiskIDCheck).WalkDir()
       1: cmd/metacache-set.go:905:cmd.listPathRaw.func2()
MinIO Object Storage Server
Copyright: 2015-2022 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2022-11-17T23-20-09Z (go1.19.3 linux/amd64)

Status:         1 Online, 0 Offline. 
API: https://ml-minio-rs-node1-master.app.warta.pl 
Console: http://10.131.3.129:9001 http://127.0.0.1:9001   

Documentation: https://min.io/docs/minio/linux/index.html

API: SYSTEM()
Time: 08:04:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:04:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:04:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:05:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:05:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:05:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:06:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:06:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:06:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage.json) returned an error (file access denied) (*fmt.wrapError)
      12: internal/logger/logger.go:258:logger.LogIf()
      11: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      10: internal/logger/logonce.go:135:logger.LogOnceIf()
       9: cmd/erasure-object.go:555:cmd.readAllXL()
       8: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       7: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       6: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
       5: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       4: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       3: cmd/config-common.go:59:cmd.readConfig()
       2: cmd/data-usage.go:95:cmd.loadDataUsageFromBackend()
       1: cmd/bucket-replication.go:3070:cmd.(*ReplicationPool).saveStatsToDisk()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:07:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:08:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:08:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:08:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:09:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:09:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:09:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:10:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:10:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:10:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:11:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:11:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:11:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: open /data/.minio.sys/config/iam/format.json/xl.meta: operation not permitted (*fs.PathError)
       5: internal/logger/logger.go:258:logger.LogIf()
       4: cmd/metacache-walk.go:327:cmd.(*xlStorage).WalkDir.func2()
       3: cmd/metacache-walk.go:351:cmd.(*xlStorage).WalkDir()
       2: cmd/metacache-walk.go:361:cmd.(*xlStorageDiskIDCheck).WalkDir()
       1: cmd/metacache-set.go:905:cmd.listPathRaw.func2()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage.json) returned an error (file access denied) (*fmt.wrapError)
      12: internal/logger/logger.go:258:logger.LogIf()
      11: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      10: internal/logger/logonce.go:135:logger.LogOnceIf()
       9: cmd/erasure-object.go:555:cmd.readAllXL()
       8: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       7: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       6: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
       5: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       4: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       3: cmd/config-common.go:59:cmd.readConfig()
       2: cmd/data-usage.go:95:cmd.loadDataUsageFromBackend()
       1: cmd/bucket-replication.go:3070:cmd.(*ReplicationPool).saveStatsToDisk()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:12:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:13:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:13:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:13:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:14:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:14:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:14:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:15:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:15:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:15:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:16:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:16:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:16:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage.json) returned an error (file access denied) (*fmt.wrapError)
      12: internal/logger/logger.go:258:logger.LogIf()
      11: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
      10: internal/logger/logonce.go:135:logger.LogOnceIf()
       9: cmd/erasure-object.go:555:cmd.readAllXL()
       8: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       7: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       6: cmd/erasure-sets.go:934:cmd.(*erasureSets).GetObjectNInfo()
       5: cmd/erasure-server-pool.go:791:cmd.(*erasureServerPools).GetObjectNInfo()
       4: cmd/config-common.go:38:cmd.readConfigWithMetadata()
       3: cmd/config-common.go:59:cmd.readConfig()
       2: cmd/data-usage.go:95:cmd.loadDataUsageFromBackend()
       1: cmd/bucket-replication.go:3070:cmd.(*ReplicationPool).saveStatsToDisk()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Drive /data, path (.minio.sys/buckets/.usage-cache.bin) returned an error (file access denied) (*fmt.wrapError)
       9: internal/logger/logger.go:258:logger.LogIf()
       8: internal/logger/logonce.go:104:logger.(*logOnceType).logOnceIf()
       7: internal/logger/logonce.go:135:logger.LogOnceIf()
       6: cmd/erasure-object.go:555:cmd.readAllXL()
       5: cmd/erasure-object.go:623:cmd.erasureObjects.getObjectFileInfo()
       4: cmd/erasure-object.go:221:cmd.erasureObjects.GetObjectNInfo()
       3: cmd/data-usage-cache.go:859:cmd.(*dataUsageCache).load()
       2: cmd/erasure.go:368:cmd.erasureObjects.nsScanner()
       1: cmd/erasure-server-pool.go:651:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       2: internal/logger/logger.go:258:logger.LogIf()
       1: cmd/erasure-server-pool.go:653:cmd.(*erasureServerPools).NSScanner.func2()

API: SYSTEM()
Time: 08:17:05 UTC 11/25/2022
DeploymentID: 6401d6db-45d6-4c9e-8e52-c044bf11793f
Error: Prefix access is denied: .minio.sys/buckets/.usage-cache.bin (cmd.PrefixAccessDenied)
       3: internal/logger/logger.go:258:logger.LogIf()
       2: cmd/data-scanner.go:209:cmd.runDataScanner()
       1: cmd/data-scanner.go:78:cmd.initDataScanner.func1()

[harshavardhana]

You simply re-used the previous drive that is not how migration works @mirekphd - you need to migrate to a fresh instance of MinIO. Please read https://min.io/docs/minio/linux/operations/install-deploy-manage/migrate-fs-gateway.html

[mirekphd]

Note that I have not necessarily re-used the "drive" [1]. Some of the "drives" I have this issue with, are PVCs in the "ab" namespace which were completely untouched before (unless they were re-cycled due to incorrect PV recycling policy, which is rare), should be pristine and both of them were indeed totally empty, created a few days ago for a new user.

[1] (not a drive as such - it's a PVC on a NFS server run on localhost in a k8s clone Openshift. I suppose the only non-standard feature distinguishing our (and other OKD / Openshift users) setup from the typical hard drive is a difference between the container user ID and owner of the files / folders - user nobody (as usual for NFS root-squash setups). The container ID which the MinIO server uses at runtime is a high ID randomly imposed at container startup - but we can pin the ID to fix it forever, however not to a value we choose, like the typical 1000 or 1001, it is necessarily a value that must be chosen from a different range. However, MinIO has no problem creating its ~/.minio.sys config folder in all these scenarios, only it later becomes write-protected, as seen from the logs. As for the reason why the files are not listed, only buckets, I can only speculate that the listing permission may be separate for buckets and for files.

As a side note, the last working version of MinIO is one day newer, mirekphd/ml-minio22:20221022, not the one tagged 20221021 (both of these working containers should correspond to the official minio/minio container daily tags). For an example of a non-working container affected by this config access issue see: mirekphd/ml-minio22:20221124.