Failed to upload the competition bundle when using Minio.

[160610101]

Hi there,
I tried to set up Minio storage and the codalab instance on my server. I follow the structure https://github.com/codalab/codalab-competitions/wiki/Minio-storage and finish setting policies using minio's "mc" helper. Now I can visit the codalab but I cannot upload my competiton bundlle. "Sorry, failed to upload the file" it mentioned.
Error reports in the codalab browser but nothing reports in the minio browser. And there are also no errors reported on the back end of the server. So I don't know what the problem is。
Here are some infomation about the instances I ran.

The codalab url: http://119.3.178.155/
The Minio url: http://119.3.178.155:9000
MINIO_ACCESS_KEY:admin
MINIO_SECRET_KEY:admin123456

part of the .env of my instance:
DEFAULT_FILE_STORAGE=storages.backends.s3boto.S3BotoStorage
AWS_ACCESS_KEY_ID=admin
AWS_SECRET_ACCESS_KEY=admin123456
AWS_STORAGE_BUCKET_NAME=compl-public
AWS_STORAGE_PRIVATE_BUCKET_NAME=compl-private
AWS_S3_CALLING_FORMAT=boto.s3.connection.OrdinaryCallingFormat
AWS_S3_HOST=119.3.178.155:9000
AWS_QUERYSTRING_AUTH=False
S3DIRECT_REGION=us-east-1
S3_USE_SIGV4=True

Thank you a lot!

[Didayolo]

Hi,

• Is the AWS part of the .env file connected?
• Did you create the compl-public and compl-private buckets on Minio?
• Did you check the logs of your Codalab instance after having started it? (docker-compose logs -f)
• Did you restart Codalab after making changes in the .env file? (docker-compose down then docker-compose up -d)

[160610101]

Hi,

* Is the AWS part of the `.env` file connected?

* Did you create the `compl-public` and `compl-private` buckets on Minio?

* Did you check the logs of your Codalab instance after having started it? (`docker-compose logs -f`)

* Did you restart Codalab after making changes in the `.env` file? (`docker-compose down` then `docker-compose up -d`)

Thanks for your reply.

• Is the AWS part of the .env file connected? - I just comment the AWS part of the .env file, and uncomment the Minio part while editing it as I described above. I don’t know if this is correct。
• Did you create the compl-public and compl-private buckets on Minio? - Yes. I created these two buckets on the Minio browser, but it didn't seem to be able to do anything else. Is it enough to just create a bucket at this step? The Minio Browser can be visited with public network link http://119.3.178.155:9000 and the secrect key has been mentioned above.
• Did you check the logs of your Codalab instance after having started it? (docker-compose logs -f) - Yes.
• Did you restart Codalab after making changes in the .env file? (docker-compose down then docker-compose up -d) - Yes

Thank you a lot!

[160610101]

@Didayolo

• I used the python 3.8 development environment and cloned the latest version of codelab.

• Do I have to configure SSL certificates ?

[lrzpellegrini]

Hi there,
I have the very same problem. I'm not exactly a django expert, but it seems that the browser first asks the s3direct enpoint for the MinIO url to upload the zip file to, then tries to use that url to upload the zip.

For some reason s3direct forces "https" into the MinIO url which triggers an SSL error (tested on both Firefox and Opera by analyzing the network requests).

I tried adding the following lines to .env (172.23.0.2 is the MinIO docker instance):

AWS_S3_SECURE_URLS=False
AWS_S3_USE_SSL=False
AWS_S3_URL_PROTOCOL=http
AWS_S3_ENDPOINT_URL=http://172.23.0.2:80

and then capture them in codalab/codalab/settings/base.py as follows:

AWS_S3_SECURE_URLS = False # strtobool(os.environ.get('AWS_S3_SECURE_URLS', "True"))
AWS_S3_URL_PROTOCOL = 'http'
AWS_S3_ENDPOINT_URL = os.environ.get('AWS_S3_ENDPOINT_URL', None)
# Prints actually show the correct values from .env
print("AWS_S3_SECURE_URLS", AWS_S3_SECURE_URLS)
print('AWS_S3_URL_PROTOCOL', AWS_S3_URL_PROTOCOL)
print('AWS_S3_ENDPOINT_URL', AWS_S3_ENDPOINT_URL)

But this doesn't fix the issue.

I'm running everything using docker-compose (I created the MinIO docker container separately, I can access its interface via browser, "private" and "public" buckets created and configured).

[Didayolo]

We are currently trying to improve the documentation for local installation.
Your issue may be related with SSL.

[rcannood]

@160610101 @lrzpellegrini Did you manage to get it to work with a local minio without SSL?

@Didayolo Has the documentation already been updated? Would it be possible to provide a docker compose that also includes a local minio setup? That would make things a lot easier for debugging competition bundles without overloading the codalab production server.

[lrzpellegrini]

I successfully managed to get it to work, but I had to change many parts of the code. Moreover, I had to use a separate DNS name for MinIO. It's an erratic patchwork and I don't think that my solution would work in general.

[160610101]

@160610101 @lrzpellegrini Did you manage to get it to work with a local minio without SSL?

@Didayolo Has the documentation already been updated? Would it be possible to provide a docker compose that also includes a local minio setup? That would make things a lot easier for debugging competition bundles without overloading the codalab production server.

Sorry, it still puzzles us, we still haven't solved this problem.