[not our bug] Receiving Bad Request when calling MINIO Server from minio/api.py (

[lmcdasm]

Hello.

I am working with @vsoch on an SREGISTRY integration that uses MINIO for the storage backend with HTTPS/SSL enabled.

The environment (for testing/development) is a couple containers on a single host and looks like this:

Where the GREEN Container is our running instance of MINIO and the Two containers in purple are our MINIO clients that implement / instantiate two instances of MinioClient on startup (as i will outline below)

A bit about our minio installation

Here is the docker-compose that is in use - you can see we consuming the .mino-env on startup

• we also have done the appropriate SSL certificate (with the .PEM consisting our of cert+intermediate and the key the key)
• "know" this works, since we can see the following when we do a a simple "curl" from linux (since a browser test might be helped by baked in intermediates) we get the following (i have redacted some of our personal info)

When we login to the MINIO Web UI, we can see the following:

• we see that SSL/HTTP is working (Browser wise) (UI-pic1 below)
• we are able to see that we can create new Containers and they are written back down the backing filesystem (UI-pic2-1, UI-PIC2-2, and TERM-SHOT-1 of the filesystem after i created)

UI-PIC-1

UI-PIC-2-1

UI-PIC-2-2

TERM-SHOT-1

As well, here is the "mc config ls" output to show our setup defined

MC-LS-1

So we are pretty sure that the HTTPS action to/from MINIO is 'ok" from a certificate point of view.

In the SREGISTRY instantion of the two minio clients, we have the following code bit (and this is where we are seeing the "BadRequest failure " - note that we create two clients, one for External Access - which is HTTPS://ourfdqn as shown and there is another that is to use the internal link - minio - which would resolve on the docker "links" (we are not using a seperate brigde as docker in the docker compose above to link the client containers to the minio container). the idea being that the sregistries are trusted and thus would not implement https. However, it seems that with the secure=TRUE/FALSE, flag we end up in a situation where minio is bringing up a endpoint called https://127.0.0.1:9000.

[root@grid-singregistry-master sregistry]# docker logs -f 401a3d72a310
Attempting rotation of encrypted config, IAM users and policies on MinIO with newly supplied credentials
Rotation complete, please make sure to unset MINIO_ACCESS_KEY_OLD and MINIO_SECRET_KEY_OLD envs
Endpoint: https://172.17.0.4:9000 https://127.0.0.1:9000

The issue here is that for me to "make this work" i would need to put a SAN in our cert for 127.0.0.1.

CLIENT-CODE-SNIP-1

CLIENT-STACK-TRACE-1

From what i have understood, is that when we instantiate the minioClient call( either the internal or the External one) there is a connection, so im a bit stumped as to why this is failed with BadRequest on the if not makebucket command.

You will notice above that the exception shows a minioExternalClient call, that is from me trying both types of clients to see if there is anything instantiated.

I did a TCPDUMP on port 9000 on the NIC that hold the docker bridge and container traffic and no request on port 9000 ever arrived during any startup, so i suspect that the instantiation is failing of the client, but im not sure how to further in minio.py figure out if its cause of a SSL/non SSL mix up (our idea is that all communications shall be over HTTPS (internal or external as we start to distribute it).

Perhaps a silly configuration i am missing with the secure flag for the clients?

Any help would be much appreciated.
D

[vsoch]

This is beautifully documented @lmcdasm ! And we can also provide detail that without ssl (just http), it works without a hitch. And @lmcdasm can you please comment if you tried keeping the internal client to just use http (and if it worked?)

[lmcdasm]

Hello

@vsoch - will try to address your points as well and illustrate.

To continue with the Thread of the current "makeBucket call that throws a bad request" and since i can create a bucket in the MINIO BACKEND, i decided to comment out that call and test it. in this way, we can see that the components start up (since there is no "usage") of the call to the MINIO endpoint of yet.. For clarity sake, i show the same code call as above, but simply commented out.

The reason that i do this is when i then do a singularity "push" - we can see when the minio client is called that i get an exception of ACCESS/SECRET KEY mismatch - so this, in the end we (@vsoch and I) think that the issue lies is that the make_bucket is failing cause there is a failed url call, and while its not hte URL persay, i had seen in earlier configuration attempts that if you are going to run an internal endpoint with https://127.0.0.1 then you have to embed a SAN into your certificate - which i could do, but not really ideal in production based environments.

I put this here for illustration - the initial call on the internal client (which is a check both for connectivity and the existence of the bucket) is a good one and "should" by all rights work to my eyes, however i think either i have to have a fqdn (internal versus external) on the internal minio endpoint in docker layout if its going to work.

At the same time, i tried with doing the "make_bucket" on the external client - which should have work, and that throws the same error as well

CODE-SNIP-COMMENT-OUT-MAKE-BUCKET

Thus from when we run our "singularity push" - which hits our sregisty app and then minio essentially we can see we get to the point of the pushing.

and in our client to minio log, we can see that instead of Badrequest, we throw a ACCESS/SECRET key error

this seems somehow related to something i saw with MC and the fact that you do a Timer on a SIGNED request somewhere internally? i could be way off base, but during my diggings, it seems a bit strange - again, i never see any traffic arrive at port 9000

I did have the thought of adding the endpoint via mc however, here is where i allude do you need to have a SAN with that IP and putting a "cert" on an IP is sorta weird - perhaps i simply need two certs and internal and an external - which is fine, i would guess i can just have the chains in the private and key parts (two certs and keys baked in or maybe there is only one FQDN / Cert+KEY pair allowed, in which case, im sure i could figure somethign out.

now - to @vsoch request, I will roll the HTTP (no ssl) equivalent setup and post for comparisons and hopefully that will help out. - give me a hour or so to roll it out and ill re-post the same bits.

Cheers,
D

[vsoch]

@lmcdasm the call that is failing is not an interaction between the external client and server (although your push originally triggers it) but rather it is the internal client requesting to generate a signed url (to return to the external client). This call fails, which is why you don't see any traffic at port 9000.

[lmcdasm]

@lmcdasm the call that is failing is not an interaction between the external client and server (although your push originally triggers it) but rather it is the internal client requesting to generate a signed url (to return to the external client). This call fails, which is why you don't see any traffic at port 9000.

excellent - that could the thread to pull on here.. as i said, in some Minio documentation that there is a part to set an unsigned call - however when i do the mc add part and provide the key (myminio) i thought this would do it..

[vsoch]

@lmcdasm the url is generated here and note that there is an argument for any extra headers that are needed. I think a solid way to start debugging would be to generate this URL for a working call (with http) and then one with https (not working).

• the multipart upload is originally requested here (this generates an upload id we can return to the external client
• then the PUT request generates the signed URL just for each blob here

So basically: we POST from the external client to the server to request the multipart. Then the internal client starts the request to generate an upload id (and other metadata) that is returned to the external client. The external client then uses the upload ID for each chunk, doing a PUT to request the URL where to put the chunk. So likely the current setup is failing at the original request (the one to generate the upload id) which comes down to:

res = s3.create_multipart_upload(Bucket=MINIO_BUCKET, Key=storage)

[vsoch]

And keep trying @lmcdasm - you're doing great, detailed work, and figuring this out is going to be hugely useful for the community! Don't worry if it takes some time, it took me ~1.5 months to get the original implementation working.

[lmcdasm]

And keep trying @lmcdasm - you're doing great, detailed work, and figuring this out is going to be hugely useful for the community! Don't worry if it takes some time, it took me ~1.5 months to get the original implementation working.

Thanks.. the one thing in the end is the docker vs. the long term K8s deployment play.. in our world, we have ingress controllers (that will take the place of nginx) and we will use a service mesh in k8s for certificate management.. in that sense, we wouldnt have a "internal and external client" - there would be "one call" to any publish endpoint (since they are all backed by a HTTPS ingress controller that is cluster-wide">.

i did notice that there is a context switch.. int he make_bucket at the start, you are using the minioClient (internal), however for the presign call you are switch back out to the HTTPS,( i tried the makebucket with the ExternalClient object as well and got the same result), but it was something i noticed, perhaps we need to add a check at instantiation time that both endpoints are up and working?

will keep at it.. had some family time and will post the http flavour shortly.

[vsoch]

I didn't stop / exit from the server in case it didn't work because it would be shown in the logs and subsequent failure, and also it would make it harder to debug if I exit immediately. But I definitely agree maybe there could be some way to better show that.

[lmcdasm]

One thing i did notice as well (from the external client side, i think we are getting as far as URL generation, since i see that the scs-library (Singuliary) apiGet - they are getting the hash that you generate and attempting to make the post to that URL

postFileV2 below

DEBUG [U=0,P=60094] apiGet() apiGet calling v1/entities/daniel.smith
DEBUG [U=0,P=60094] apiGet() apiGet calling v1/collections/daniel.smith/dasm-test
DEBUG [U=0,P=60094] apiGet() apiGet calling v1/containers/daniel.smith/dasm-test/something

DASM This is i assume the registry given the hash for the container to be uploaded

DEBUG [U=0,P=60094] apiGet() apiGet calling v1/images/daniel.smith/dasm-test/something:sha256.b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101?arch=amd64
DEBUG [U=0,P=60094] postFileWrapper() Now uploading to the library
DEBUG [U=0,P=60094] postFileV2() Using legacy (single part) uploader

DASM - here singularity is using their postMethod to that URL
DEBUG [U=0,P=60094] legacyPostFileV2() legacyPostFileV2 calling v2/imagefile/15
DASM - here they try to post calling the endpoint that was provided back - notice that it uses and endpoint of v2/imagefile/15
is that an obfuscation back to the daniel.smith/dasm-test/something:sha256 ?

Thanks alot for all the input so far (working on a little stub now for the working vs non-working call to the URL).
D

[vsoch]

I don't see the PUT request there - I see a request that goes to the legacy uploader, which isn't multipart (and is not Minio).

[lmcdasm]

I don't see the PUT request there - I see a request that goes to the legacy uploader, which isn't multipart (and is not Minio).

hmm. im not sure i understand.. this is singularity in DEBUG calling.> So its configured to use SREGISTRY as the remote.

We can see in the UWSGI logs the following
SINGULARITY CALLS:
DEBUG [U=0,P=60094] apiGet() apiGet calling v1/entities/daniel.smith

This INVOKES iN REGISTRY the following method
GET NamedEntityView
which replies back with /v1/entities/daniel.smith

then we see that SINGULARITY (this is the SCS-LIBRARY that makes up the SIngularity Container Storage call as part of singularity) send the next apiGet() call
DEBUG [U=0,P=82236] apiGet() apiGet calling v1/collections/daniel.smith/dasm-test

Which maps to SREG call of
GET GetNamedCollectionView

then we see that SINGUARLITY CLIENT soes the
apiGet calling v1/containers/daniel.smith/dasm-test/something

And that maps to
GET GetNamedContainerView

We see then that SINGUARLIT getApi() does the following call
apiGet calling v1/images/daniel.smith/dasm-test/something:sha256.b226c379e19853f16f1dc76b13b70deaf822afd6078c5bc3cd9b921d32cf7101?arch=amd64

in the SREG logs, this generates the folowing method
GET PushNamedContainerView

Then we see the SINGULARITY try to push it:

DEBUG [U=0,P=82236] postFileWrapper() Now uploading to the library
DEBUG [U=0,P=82236] postFileV2() Using legacy (single part) uploader
DEBUG [U=0,P=82236] legacyPostFileV2() legacyPostFileV2 calling v2/imagefile/17
DEBUG [U=0,P=82236] apiCreate() apiCreate calling v2/imagefile/17

and this can only have been provided from the apiGet() calls above, in the REG log this method is called

[pid: 59|app: 0|req: 5/7] 10.179.111.4 () {34 vars in 453 bytes} [Thu Jul 30 16:47:51 2020] GET /version => generated 58 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 0)
POST RequestPushImageFileView

_Thus at this point, im not sure what you mean, but you "dont see a PUT" there, and that it goes to a lergacy uploader - from the external client point of view - its using the URL that is provided back to it? so im not sure how it (singularity) could use anything else that what is provided in the answers "back" to singularity?? This is just a simple "push" to a remote from command line, so im lost when you say "its using a legacy uploader" - reading the scs-library, it uses whatever path is provided in the previous call from the server (registry) side if i follow -

sorry if im not getting it