MINIO + HTTPS - Unclear how you performed HTTPS without having IP SANs in our cert / using external Cert, perhaps the documentation could be updated to show it working with external cert? #320
Comments
|
I haven set up the server with https, but I suspect you might need to configure hostnames or write the config so it uses the correct internal / external addresses (or maybe have two)? Did you see the link to https://docs.min.io/docs/how-to-secure-access-to-minio-server-with-tls.html? That’s what I pointed the user to in the current docs, and I was hoping you would be able to follow steps there to set it up. I can’t easily locally reproduce with a custom domain, etc. so it’s definitely challenging to help via that method! |
|
A quick observations:
That would be a good thing to try, but that address would need to be defined in the hostname to point to the container (which currently is named "minio"
at first I thought was concerning, but probably is just refencing the definition of the previous envars. You might try following that advice to unset and update.
What I would try to do is first look into other examples of inter-container communication and https, and see if some custom work is needed. I've never had to do anything special, but when you add in custom networking it tends to complicate things. You might also try breaking it down into the simplest example to reproduce (e.g., connecting via an internal/external client and trying to connect to a bucket) and then ask the folks at Minio (e.g., the miniopy repo) specificaly about the differences in setting up the cert between an internal and external client. |
|
More things to try! Via this thread: minio/minio-py#759 (comment) it looks like you might try making the internal url http, and also playing around with the secure=True/False variable when you instantiate the call. |
|
Hello Again.. thanks for the quick answers.. So essentially i was at those links and i had gotten to the point of the following:
Along the above lines, i can "get that to work sort of", however this means that in the links definitions, i have to use a 'minio.mydomain.net" for example and then perpetuate that through the docker compose - trying that out.
have rotation the credentials a couple times on restarts in the past CI build loops
Thats a great idea above - thanks! i have installed s3 and MC and the "external container" link seems alright.. i think its getting HTTPS on the internal link by service name (in a docker-compose context). i am thinking i can probably just do the docker build of the minio and drop in the hostname and others directly.. For the certificate, since i have a valid one ( you can see the HTTPS is working in the UI, and i can inspect the cert on the minio browser/site) im pretty sure the mapping to the cert/private is correct and its a wildcard off our domain, so will server anything top level). I think its a question of that mapping of the endpoint name back in and the MINIO_SSL TRue flag |
Excellent! - thanks, seems like its the path im on.. Question - is there a place in docs to post reference example configurations and layouts/samples in the project (anonymized).? it might be neat to have some pictures and examples if we get this working (ideally https on the in and outside), or http/https ... could help others maybe? Cheers |
|
That’s a fantastic idea! Right now we have Minio under storage under setup, so we could either add more content there (or if there is enough to warrant separation) make another link. We also could make a section under deployments (right now there is just an old ansible recipe.) A combination of the two would be having the basic setup where it is now, and linking to a more detailed “Minio with HTTPS Deployment” page under deployments. I can definitely help with this however you need - I am so grateful that you are testing this out and paving the way for future users! |
|
Hey there.. ok.. i have a lazy guide that is basically 'from Centos to running install on docker" (pre-cursor to making some k8s manifests), and happy to contribute that with some screen shots and such (once i sort this out)> for the issue at hand, going into the python stack trace, we can see there are two MinioClients that are defined (minioClient and minioClientExternal). and i see that you then create a session and s3 client connection for each Minio that you stand up, looks good. However, line 62 seems (maybe to me) out of place, since you are doing a "make_bucket call" before you have a session, thus you end up with a tuple like this "host: {minio] url: /testbucket/" (which would actually work with your http:// if MINIO_SSL is False prefix + MINIO_DOMAIN_NAME - however, (in my case, im still stuck with an FQDN issue - however, there is a "argument" that could be made that since its "internal communication" the use of a signed CA cert (digicert in my case) is overkill for the internal link path) - However, when i commented out that 'make_bucket" and rebuilt locally with the following shub/settings/config.py and the call commented out shown below, SRegistry is now up, and working (weird huh? :P). What i suspect (and sort of intuiting from the minio() constructor is that in the case of your mark_bucket, you dont have a session, so even though its http (to minio) you still need credentials, and results in the scheduler and worker throwing a Bad request. ... all of this is a bit of guess and "history | grep build | wc -l " (docker build) = 104 and the registry is up now with https :) shub/apps/library/views/minio.py (in your flask setup) Here is my shub/settings/config.py presently: MINIO_SERVER = "minio:9000" # Internal to sregistry Here is what my "mc config ls " looks like. ./mc config host ls local myminio play s3 |
|
Oh and to add, i have it working now with LDAPs (with with a baked Private CA cert, since internally we dont want to have to have linux clients need a standard SSL cert (A Private CA is fine) have a meeting tomorrow to do with the SAML/SSO (sorting out iDP and SP configuration and the metadata.xml to do it). The introduction of Minio is a really great idea, there are somethings i would like to add (help to add).
Shoot me the branch in the repo where you want pushs to (docs directly? .RST, .MD , guidelines?) where i can post a "Use Case Example Tutorial - Single Host/Docker/External Owned Cert" or some title can be butchers, and i throw up a diagram and lazy guide/screens and we can edit from there)" Continuing to test, but so far Logins, Teams, Collections, AD LDAP looking good! |
|
so.. it would seem im back to something "familiar" the client: singularity remote use company-remote API Key is "valid" Steps of Issue: and in the uswgi here is what i am seeing: USWGI LOG: od.json => generated 561 bytes in 10 msecs (HTTP/1.1 200) 3 headers in 109 bytes (1 switches on core 2) CLIENT CALL IN DEBUG: [root@ip-0AB36F04 ~]# singularity --debug push -U dasm-test.sif library://daniel.smith/mytest/sometthing:latest So, this leads me to the SWAGGER Page to test the apiCreate Call - i suspect that the "check if there, if not create is faling" i looked on the API page (swagger) for the registry and i dont see you have a /v1/apiCreate call available there. looking more into the code call now - i suspect it is cause the singularity client is calling that create v1/entities/daniel.smith and since its not exposed via your flask endpoints it fails maybe? I put some more debug statements and the issue was the token in validate_token was returning false so not found. This is due to the fact that the rebuilds, need a new token for the client each time (duh!) ... |
|
Hey there. i have essentially got it working, however i made a mistake with the .minio-env and the order of passwords is mixed and messed. do i go back to the original default creds and start the chain again? as you can see in the client DEBUG, im "almost there", however due to the miino ACCESS/KEY SECRET/KEY mixup the /v2/call is failing presently. [root@ip-0AB36F04 ~]# singularity --debug push -U dasm-test.sif library://daniel.smith/my-test/sometthing:latest This is cause Minio is out now, it was throwing the MINIM TIMEOUT For SIgned URL and i was about to increase that to a high value. And the fix for this is to remove the _OLD from the .minio-env file and restart, so that you dont double encrypt and it is up again |
|
hey @lmcdasm! Sorry for the delay in response, I did in fact go to sleep! I've put together a branch and a page for you to work on documentation: https://github.com/singularityhub/sregistry/blob/add/minio-ssl-docs/docs/_docs/deploy/minio-ssl.md Specifically, that's linked from the sidebar under the deployment section, and also linked from the Storage setup documentation (and it links back to it). For images, you can either put them in assets/img/minio (or similar) or just an img/ folder in the same folder with the file, and then the path would be relative (actually one level up) from the page. E.g.,, with this structure: In the markdown the image would be linked like: Now to answer your questions!
Just to be clear - the minio client that creates the bucket is authenticated Also, the server is not implemented in Flask, it's in Django - the "framework for perfectionists!" It's probably my favorite :)
|
|
I think it would be worth getting Minio working, and if you can come up with a basic use case of one of the clients having a permission error to connect to the bucket, and then showing the mc debug output to the minio team, they could help you out. Remember that you can shell into the docker container, then do |
This (below) is amazing, i will start to .md up my stuff and push as i write.
i believe that somewhere in the MINIO installation part there was something to add for 7 minutes or something.. on successive builds/restarts - i suspect that this might have expired? is that mc call to add a signed endpoint to minio (myminio in the docs) you are meaning in the CSG (i havent had a chance to look yet so if yet, you can leave me to find it :P)
OK.. so.. i will show a bit of the hand at the architecture here discussion point - on a VM/Node base abstraction layer environment:
Discussion - take it a bit further into K8S: massive oversimplification of the work needed... :)
DASM - yes, however i do have two certs and can do a internal and external without an issue if need be. as for why would it be needed, there is the idea that today you use nginx as your "ingress controller" essentially and this is fine and can port nicely to k8s . however, when we want to talk in the realm of service registry and Service mesh, where there is a service that provides a common ingress that all apps in a cluster bind their FQDNs to this gateway (and thus certificate management at a deployment FQDN level is vastly improved since its one cert (or many) for the cluster ). in this case the ngnix front door will go and at that point the idea of external client/internal client might become "irrelevant" - and even desired since it would make the idea of "many replicas of sregistry components" (minio_1, minio_2,minio_3 - running in a given name space with a Service name of "minio" but now cluster redundant - and with HPFS redundant parallel storage) we know have something very robust and geo-distribution ready.
DASM - yup, worked out the details, once im back back to a working rig ( i think i just botched up Minioa a bit with trying things), im going to start on that integration.> For the docs, we can show a "LDAP reference Configuration' and " SSO/SAML iDp" one as well/ varient
DASM - is not super pressing, we use a couple different Cloud Providers
Writing this, the use case of having instances of sregistry deployed globally and rather than using a disk/network fil PVC in k8s, that when SREG is deployed it can be configured to use many different Cloud Provide Object storage (fast and cheap) ) - maybe a "feature request"
Action plan at this point, have lunch :P hopefully i didnt ramble too much.. not enough coffee yet. |
|
Hey there. So we are back in business, MINIO is up, HTTPS is working, i can login an i can create Collections and such> when im doing the following Singularity command (and i have updated my .sregistry and i coped the new token in after loging in ) for my singularity client, see below CLIENT SIDE: UPDATE KEY: WHICH ONE IS IN USE: [root@ip-0AB36F04 ~]# singularity remote status SINGULARITY PUSH (with debug turned on): [root@ip-0AB36F04 ~]# singularity --debug push -U dasm-test.sif library://daniel.smith/dasm-test/sometthing:latest HERE IS THE USGW LOG FOR THAT PUSH - you can see its giving an Access Denied ytes in 9 msecs (HTTP/1.1 200) 3 headers in 109 bytes (1 switches on core 2) Thus its not "me" that is chosing to use that endpoint - im looking in the singuliarity API to see how to force it to use a configured endpoint (or to override it). Cheers, |
|
Thanks for the details! A lot of it goes over my head, but I'm definitely here to learn and help you as much as I can!
I'm not sure about what you are referring to here, at least without https I've been able to restart the containers without any issue.
For the library endpoint you don't need the .sregistry file, that's for an older push without it.
This is an issue with the key/secret for the internal client. What do you mean it's not you choosing to use it? As I suggested before, I think the core of this issue is with the ssl setup, because it works without a hitch with http.
Indeed! Singularity Hub (different but similar) has all storage interactions directly from Google Storage, which basically doesn't stress the host running it - that host just generates a signed URL and sends it off! :) |
|
A bit more of an update We can see that the MINIO Access Key error is still being thrown. as seen in this pic:
if we go in the collection we see they the builds are not really there though (and this makes sense based on the client debug, since the call to create the container label and hash and such all happens before hand).. All good>
When i reanable your make bucket call, it falls with this call to host/testbucket and i dont get any futher.. During handling of the above exception, another exception occurred: Traceback (most recent call last): |
|
You are trying to debug layers of errors, but the core issue is that the make bucket command is failing. As I already suggested, you should interactively produce this error, and provide the details of your SSL setup to the minio crew and get feedback. You of course aren't going to have anything upstream working if the basic initial connection to create a bucket doesn't work. Feel free to cc me on the minio issue, thanks! |
|
heya.. yup.. im working through it now. and will let you know (update ticket here and upstream) once i understand that part. Thanks again, will keep ya posted on things.. its not "that far away" D |
|
Ok.. so to recap where we are at (sorry for the delay, i had to find a machine with 15G swap to setup the image) Having put the credentials right into the minioClient and minioExternalClient calls, seems to have got things working end to end when its legacy, but below you see i get a new error with "missing credentials". and now im to the res = s3.create line that you outlined, so im doing some debugging again in images.py.. something is getting lost in the credential passing, since they are coded into minio.py, so its the use of the MINIO_ACCESS and MINIO_SECRET - when i added the creds right into the session instantiation i think it will work (building now, will know shortly). for the test of large files, here is what i have done>
My SING DEF:Bootstrap: docker %setup %files %environment %post %runscript %startscript %test %labels %help MY CONTAINER :[root@ip-0AB36F04 ~]# ls -tlr Found a token in validdate_token() THE SINGULARITY (EXTERNAL - you can see tries to use the MULTIPART) call@ip-0AB36F04 ~]# singularity --debug push -U bigcontainer.sif library://daniel.smith/dasm-test/biggie:latest |
|
Look at the core error, it's unable to find credentials. This seems like a new bug, so likely your environment variables aren't getting found for the Minio keys. The session that uses them is here . |
|
Hey there. So, once i got the credentials, what i saw is that the Switch for MINIO_SSL in the s3.client part (just below session creation where you have the keys) was sending it to http://minio:9000 and this generated and error (Connection closed by peer). im fixing it so its all https through and through and should see shortly. closer and closer - but i think you're right, something is going on in the environment - im working my way backwards to that point |
|
and thanks so much for you patience as i step through this... gonna feel dumb when i find it, i think :P |
|
I’m going to feel elated that we figured it out! Well mostly you, I’m here for moral support :) I’m going to be signing off for the evening, and I have a talk first thing tomorrow but should be around after that. Don’t forget to take some breaks on your end too! |
|
as will the 200 so grid devs that are waiting for me to launch this :) have a good night and ill keep you updated (and youll see example / tutorial code pushes as well once its working : )) |
|
And Voila, we have working action HTTPS end to end (with Verfiy enabled). there was another place at the end of the transaction minioClient.remove_object that i switched to minioExternalClient and then it worked> from what i understand, the idea was to have the calls use an internal Http(s)-less path and then external calls https - however, using the https://127.0.0.1 local definition becomes impossible and using the name 'minio' would really work. here are the logs showing that its working :) - im going to run our rest rig and give it some paces and then think about perhaps some improvements that might allow for the 4 combinations to exist with internal/external (http/https http/http, https/http Again, happy to have a working build! SREG LOG:
SINGULARITY CLIENT CALL:
Cheers! |
|
Woohoo!! What great work!! I can't wait to see it! Now we see the POST that I was talking about - the |
|
Hey any updates? I expected a PR and a month has passed, are you doing ok? |
|
Closing issue, no response. |
Hello
i tried to do the installation today and could not get the combination of components using HTTPS (which is setup correctly in NGINIX) with MINIO. Referring to this section here
https://singularityhub.github.io/sregistry/docs/install/server
As best as i can tell, you end up with a "local" endpoint (that is done and listening on the 17.16.x.y internal docker network. however, this is not possible with a "name" since a real cert (from our CA/PKI) would not have a SAN for 127.0.0.1 it it (only names).
we can see that Minio comes up and i can define it as a external server (reachable via the browser from https://mydomain.net:9000 (and SSL enabled) and i can create a bucket however (again this is a guess), since you are using the parameter here
MINIO_SERVER = "minio:9000" # Internal to sregistry -
this seems to use the docker named link and thus when you have installed a SSL cert for mydomain.net, and internally registry tried to use minio (and thus https://127.0.0.1) you get a failure.
Any idea on how i can correct this?
here is the shub/setttings/config.py
STORAGE
here i have tried some different variants, but havent had any luck
MINIO_SERVER = "singregistry.mydomain.net:9000" # Internal to sregistry
MINIO_EXTERNAL_SERVER = (
"singregistry.mydomain.net:9000" # minio server for Singularity to interact with
)
MINIO_BUCKET = "testbucket"
MINIO_SSL = False # use SSL for minio
MINIO_SIGNED_URL_EXPIRE_MINUTES = 5
MINIO_REGION = "us-east-1"
MINIO_MULTIPART_UPLOAD = True
Don't clean up images in Minio that are no longer referenced by sregistry
DISABLE_MINIO_CLEANUP = False
Here is a picture showing that i am able to get into MINIO and see the test bucket created. (testbucket)
Here is the "mc command" output (config ls)
[root@grid-singregistry-master sregistry]# !323
./mc config host ls
gcs
URL : https://storage.googleapis.com
AccessKey : YOUR-ACCESS-KEY-HERE
SecretKey : YOUR-SECRET-KEY-HERE
API : S3v2
Lookup : dns
local
URL : http://localhost:9000
AccessKey :
SecretKey :
API :
Lookup : auto
myminio
URL : https://singregistry.mydomain.net:9000
AccessKey : abc123abc
SecretKey : abc123abc123
API : s3v4
Lookup : auto
play
URL : https://play.min.io
AccessKey : Q3AM3UQ867SPQQA43P2F
SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
API : S3v4
Lookup : auto
s3
URL : https://s3.amazonaws.com
AccessKey : YOUR-ACCESS-KEY-HERE
SecretKey : YOUR-SECRET-KEY-HERE
API : S3v4
Lookup : dns
[root@grid-singregistry-master sregistry]#
but i cant see why uswgi (but its the worker and the scheduler container continue to throw this error):
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/local/lib/python3.5/site-packages/django/utils/deprecation.py", line 94, in call
response = response or self.get_response(request)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception
response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
callback, param_dict = resolver.resolve_error_handler(500)
File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 597, in resolve_error_handler
callback = getattr(self.urlconf_module, 'handler%s' % view_type, None)
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in get
res = instance.dict[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 577, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/local/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 985, in _gcd_import
File "", line 968, in _find_and_load
File "", line 957, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 697, in exec_module
File "", line 222, in _call_with_frames_removed
File "./shub/urls.py", line 19, in
from shub.apps.library import urls as library_urls
File "./shub/apps/library/urls.py", line 12, in
from shub.apps.library import views
File "./shub/apps/library/views/init.py", line 2, in
from .images import (
File "./shub/apps/library/views/images.py", line 33, in
from .minio import (
File "./shub/apps/library/views/minio.py", line 62, in
if not minioClient.bucket_exists(MINIO_BUCKET):
File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 453, in bucket_exists
self._url_open('HEAD', bucket_name=bucket_name)
File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 2080, in _url_open
object_name).get_exception()
minio.error.ResponseError: ResponseError: code: BadRequest, message: Bad Request, bucket_name: testbucket, object_name: None, request_id: , host_id: , region:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 141, in call
response = self.get_response(request)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 75, in get_response
response = self._middleware_chain(request)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception
response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 128, in handle_uncaught_exception
callback, param_dict = resolver.resolve_error_handler(500)
File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 597, in resolve_error_handler
callback = getattr(self.urlconf_module, 'handler%s' % view_type, None)
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 80, in get
res = instance.dict[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 577, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/local/lib/python3.5/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 985, in _gcd_import
File "", line 968, in _find_and_load
File "", line 957, in _find_and_load_unlocked
File "", line 673, in _load_unlocked
File "", line 697, in exec_module
File "", line 222, in _call_with_frames_removed
File "./shub/urls.py", line 19, in
from shub.apps.library import urls as library_urls
File "./shub/apps/library/urls.py", line 12, in
from shub.apps.library import views
File "./shub/apps/library/views/init.py", line 2, in
from .images import (
File "./shub/apps/library/views/images.py", line 33, in
from .minio import (
File "./shub/apps/library/views/minio.py", line 62, in
if not minioClient.bucket_exists(MINIO_BUCKET):
File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 453, in bucket_exists
self._url_open('HEAD', bucket_name=bucket_name)
File "/usr/local/lib/python3.5/site-packages/minio/api.py", line 2080, in _url_open
object_name).get_exception()
minio.error.ResponseError: ResponseError: code: BadRequest, message: Bad Request, bucket_name: testbucket, object_name: None, request_id: , host_id: , region:
[pid: 59|app: 0|req: 2/4] 10.3.103.8 () {48 vars in 972 bytes} [Tue Jul 28 22:00:28 2020] GET /favicon.ico => generated 0 bytes in 1256 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 3)
[root@grid-singregistry-master sregistry]#
running the MC TRACE< doesnt provide anything in the window and nothing shows up int he minio logs - here they are:
[root@grid-singregistry-master sregistry]# docker logs -f 3cf
Attempting rotation of encrypted config, IAM users and policies on MinIO with newly supplied credentials
Rotation complete, please make sure to unset MINIO_ACCESS_KEY_OLD and MINIO_SECRET_KEY_OLD envs
Endpoint: https://172.17.0.4:9000 https://127.0.0.1:9000
Browser Access:
https://172.17.0.4:9000 https://127.0.0.1:9000
Object API (Amazon S3 compatible):
Go: https://docs.min.io/docs/golang-client-quickstart-guide
Java: https://docs.min.io/docs/java-client-quickstart-guide
Python: https://docs.min.io/docs/python-client-quickstart-guide
JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
.NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
Any tips here would be great, since if i have to add a 127.0.0.1 SAN into our wildcard cert for our domain, it means i have to reissue it to 500 + machines.
Thanks!
The text was updated successfully, but these errors were encountered: