Upload page 503 Service Temporarily Not Available

[vathyamodot]

hi I downloaded the files and made few changes to suit our docker environment and trying to run on port 50075. The initial upload page came up and then selected the files to upload from the pop-up selection box. Then I am getting the 503 Service Temporarily Available. Do I have to make any changes or missed any changes? Here is the screenshot of the error.

[vsoch]

It’s hard for me to debug with changes I don’t know, but I’d check if you exposed the port, and made sure to update the Nginx.conf, and also verify the temporary upload folders exist in /tmp.

[vathyamodot]

hi, I think I found the issue. The sub directories under _upload are missing and had to manually create all 0 to 9 directories. The command listed was creating _upload/{0..9} directory by default. After I created subdirectories That error is vanished but running into other errors. Here is the output from nginx error.log. I did not change any of your stuff.

2020/10/27 17:31:12 [error] 6#6: *2 open() "/var/www/static/img/favicon.ico" failed (2: No such file or directory), client: 10.255.0.2, server: localhost, request: "GET /static/img/favicon.ico HTTP/1.1", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 17:32:37 [error] 6#6: *5 failed to create output file "/var/www/images/_upload/1/0000000001" for "docker-support-b1wnog3-20200403-15_38_45.zip" (2: No such file or directory), client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 17:39:43 [error] 6#6: *7 failed to create output file "/var/www/images/_upload/2/0000000002" for "docker-support-b1wnog3-20200403-15_38_45.zip" (2: No such file or directory), client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 18:28:58 [error] 6#6: *10 failed to create output file "/var/www/images/_upload/3/0000000003" for "docker-support-b1wnog3-20200403-15_38_45.zip" (2: No such file or directory), client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 18:35:38 [error] 6#6: *15 failed to create output file "/var/www/images/_upload/4/0000000004" for "docker-support-b1wnog3-20200403-15_38_45.zip" (2: No such file or directory), client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 19:21:13 [warn] 6#6: *18 an upstream response is buffered to a temporary file /var/cache/nginx/uwsgi_temp/6/00/0000000006 while reading upstream, client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", upstream: "uwsgi://10.0.32.7:3031", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 19:21:14 [error] 6#6: *18 failed to remove destination file "/var/www/images/_upload/5/0000000005" after http status 500 (2: No such file or directory) while closing request, client: 10.255.0.2, server: 0.0.0.0:80
2020/10/27 19:25:35 [warn] 6#6: *21 an upstream response is buffered to a temporary file /var/cache/nginx/uwsgi_temp/8/00/0000000008 while reading upstream, client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", upstream: "uwsgi://10.0.32.7:3031", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 19:25:36 [error] 6#6: *21 failed to remove destination file "/var/www/images/_upload/7/0000000007" after http status 500 (2: No such file or directory) while closing request, client: 10.255.0.2, server: 0.0.0.0:80
2020/10/27 19:26:31 [warn] 6#6: *21 an upstream response is buffered to a temporary file /var/cache/nginx/uwsgi_temp/0/01/0000000010 while reading upstream, client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", upstream: "uwsgi://10.0.32.7:3031", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 19:26:32 [error] 6#6: *21 failed to remove destination file "/var/www/images/_upload/9/0000000009" after http status 500 (2: No such file or directory) while closing request, client: 10.255.0.2, server: 0.0.0.0:80
2020/10/27 19:27:07 [warn] 6#6: *21 an upstream response is buffered to a temporary file /var/cache/nginx/uwsgi_temp/2/01/0000000012 while reading upstream, client: 10.255.0.2, server: localhost, request: "POST /upload HTTP/1.1", upstream: "uwsgi://10.0.32.7:3031", host: "dotiuasld01:50075", referrer: "http://dotiuasld01:50075/"
2020/10/27 19:27:08 [error] 6#6: *21 failed to remove destination file "/var/www/images/_upload/1/0000000011" after http status 500 (2: No such file or directory) while closing request, client: 10.255.0.2, server: 0.0.0.0:80

[vsoch]

Yes the instruction to create them is clearly stated in the README

https://github.com/vsoch/django-nginx-upload#usage

The second error message above suggests that you did not create the database, e.g., you didn't do the migrations. Again, I can't comment on the changes that you made, but I can say that the original version, clearly following the instructions, should work.

[vathyamodot]

the command listed below is creating the directory as it is. Not creating sub folders.

sudo mkdir -p images/_upload/{0..9} && sudo chmod 777 -R images/_upload

Also do I have to create the database manually? I don't see instructions.

[vathyamodot]

When execute docker deploy using your composefile creating 3 containers. One of them is db.

[vsoch]

the command listed below is creating the directory as it is. Not creating sub folders.

It depends on having a bash shell, if you have something different or Windows you need to adjust for your needs. The migrations are done in the run_uwsgi.sh script, which is the container entrypoint: https://github.com/vsoch/django-nginx-upload/blob/master/run_uwsgi.sh. If this script isn't hit then there is something different or erroneous about your setup.

[vathyamodot]

Thanks. I see the script running as entry point I guess from the following output.

763ce969235f django-nginx-upload_uwsgi:latest "/bin/sh -c /code/ru…" 6 minutes ago Up 6 minutes 3031/tcp fileupload_uwsgi.1.cp8hvnczcv0oz51u2fpkrzvmq

After I executed it manually inside the container it fixed the issue. Now I can see the uploaded file list. Here is the output after I executed it manually. Thanks for the pointer. Still not sure why its not executed.

root@763ce969235f:/code# ./run_uwsgi.sh
No changes detected
Operations to perform:
Apply all migrations: auth
Running migrations:
No migrations to apply.
No changes detected in app 'main'
Operations to perform:
Apply all migrations: admin, auth, contenttypes, main, sessions
Running migrations:
No migrations to apply.

0 static files copied to '/var/www/static', 121 unmodified.
[uWSGI] getting INI configuration from uwsgi.ini
Tue Oct 27 20:54:06 2020 - *** Starting uWSGI 2.0.19.1 (64bit) on [Tue Oct 27 20:54:06 2020] ***
Tue Oct 27 20:54:06 2020 - compiled with version: 4.9.2 on 26 October 2020 20:31:46
Tue Oct 27 20:54:06 2020 - os: Linux-3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 11 19:12:04 EDT 2020
Tue Oct 27 20:54:06 2020 - nodename: 763ce969235f
Tue Oct 27 20:54:06 2020 - machine: x86_64
Tue Oct 27 20:54:06 2020 - clock source: unix
Tue Oct 27 20:54:06 2020 - pcre jit disabled
Tue Oct 27 20:54:06 2020 - detected number of CPU cores: 6
Tue Oct 27 20:54:06 2020 - current working directory: /code
Tue Oct 27 20:54:06 2020 - detected binary path: /usr/local/bin/uwsgi
Tue Oct 27 20:54:06 2020 - uWSGI running as root, you can use --uid/--gid/--chroot options
Tue Oct 27 20:54:06 2020 - *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Tue Oct 27 20:54:06 2020 - chdir() to /code/
Tue Oct 27 20:54:06 2020 - your memory page size is 4096 bytes
Tue Oct 27 20:54:06 2020 - detected max file descriptor number: 1048576
Tue Oct 27 20:54:06 2020 - lock engine: pthread robust mutexes
Tue Oct 27 20:54:06 2020 - thunder lock: disabled (you can enable it with --thunder-lock)
Tue Oct 27 20:54:06 2020 - probably another instance of uWSGI is running on the same address (:3031).
Tue Oct 27 20:54:06 2020 - bind(): Address already in use [core/socket.c line 769]

[vsoch]

It's likely related to your operating system, and versions of Docker and docker-compose - I've had different results on different OS in personal experience. Glad it all worked out with the manual commands, feel free to close the issue if it's resolved.

[vathyamodot]

I will probably mention explicitly to execute as command in compose and test it out. Thanks for help. I will close it out.

[vathyamodot]

Thanks.

[vathyamodot]

Hi Vanessarauras, A quick question. The current push.py handles one file at a time. Do you have a version that handles multi-file upload?

[vsoch]

hey again! That’s really just provided as a dummy example, if you want to scale it I can imagine trying multiprocessing or similar, given using Python. If you come up with an example to supplement the current please please PR to contribute back to the repository!

[vathyamodot]

Sure. I will. But it is working like a charm. Just now I converted the compose file to use azure storage account mounts. It is working great for nginx uwsci containers. But for DB still have to use file system due to the limitation of postgres. I wanted to try multi-file upload. But I am not a web developer. I am a docker professional at ODOT.

[vsoch]

Hey be careful about responding from email, I just deleted a bunch of your personal information from your signature from the last two comments that you likely don’t want out in the public on a GitHub comment.

I’m not a web developer either and this is provided as a proof of concept for others to build on so I can only give suggestions. For smaller files it is in fact possible to save data (e.g images) directly to a binary field in the database (e.g. Postgres). I’ve done this for services like app engine that don’t support writing to a filesystem other than tmp. But when you are talking about multiple file upload there are two issues at hand - the client making multiple requests (e.g using multiprocessing) and the server being able to handle it. Since this is Nginx, it would mean scaling that, either adding more containers to the Docker compose or using something more like Kubernetes. I’m happy to give this advice but beyond that is out of scope for this repository. The challenge I had at the time was implementing this for Django, and so that’s the example implemented here and nothing more than that.