S3 Persistent storage with Dockerfile nginx+uwsgi #13

fluential · 2020-07-11T05:53:20Z

This implements

S3 persistent storage so its possible to safely start and stop container, it does restore on startup + backup after 30s delay, according to a schedule
uwsgi+nginx for much better performance

For yet better performance consider using ASGI frameworks instead of WSGI.
If you still need more +400% performance for this uwsgi+nginx flask app consider using this image https://github.com/tiangolo/meinheld-gunicorn-flask-docker 😎

…code as is easier to read as-is

+ Development on index.html

…generous

- Remove converting a string to a string?

+ Added HTTPS messages + code

…quested resource."

Enables local docker volume db store

fluential · 2020-07-11T07:41:20Z

@brentvollebregt phew, I think this one is ready now, have a look

Not needed

brentvollebregt · 2020-07-12T02:15:59Z

I understand that there is currently an issue where the database will be lost when the container is removed however, I am not a huge fan of forcing everyone to use S3. I'm much more a fan of using something like a mount and users then build on top of that however they may need to.

A potential solution would be to make all database interactions use peewee which would then allow people to point at an external database. This method would also then not force people to use SQLite.

What's your opinion on this?

fluential · 2020-07-12T06:10:06Z

@brentvollebregt the S3 backup is optional in that scenario

you are not forcing anyone, if you don't specify the keys, backup would not do anything. If there is local db it would not overwrite from s3
there is a VOLUME option provided so you can just stick to it if that is your preference, check the README how to use it

So I would say its safe to merge?

The wasabi s3 compatible storage comes with 6 USD /month 2TB including, I use this a lot for running containers around the world with different providers but keep the state across.

My state is small enough that itso ok to pull it on startup. You can understand the difference between building web stack on a persistence docker volume storage of a platform/provider or just run emphemeral containers with out-of-bounds backup anywhere :)

With just using cron you could have a simple distributed sevice 😎

But to answer your question in using peewee, certainly somethign to consider but in my opinion you would need to have a solid caching layer to avoid remote db connections as much as possible and sync into remote state occasionally (same stands for SQLite). The other option is to consider something like jsonbox or similiar but same concepts stand really.

If you want to have fun I would suggest going for https://dqlite.io/ but otherwise peewee sounds good.

brentvollebregt · 2020-07-13T09:24:19Z

Dockerfile


-COPY . /app
-RUN pip install -r /var/www/requirements.txt
+RUN chmod 755 /usr/local/bin/sq*


Is this setting permissions for more than sqlitebackup.sh?

brentvollebregt · 2020-07-13T09:27:29Z

Dockerfile


-EXPOSE 8080
+RUN cd /usr/local/bin && wget https://raw.githubusercontent.com/fluential/docker-sqlite-to-s3/master/sqlite-to-s3.sh && chmod 755 sqlite*


On README.md:106 I see you reference https://github.com/jacobtomlinson/docker-sqlite-to-s3 but then use a fork of it here? Since this fork is identical to the original repo, could we use the file from the repo referenced in the README?

Good catch, will fix

brentvollebregt · 2020-07-13T09:47:11Z

config.py

@@ -1,7 +1,7 @@
 import os

 # Location of database
-DATABASE_FILE_PATH = os.path.abspath(os.getenv('DATABASE_FILE_PATH', 'data.db'))
+DATABASE_FILE_PATH = os.path.abspath(os.getenv('DATABASE_FILE_PATH', 'data/data.db'))


The default needs to be left as 'data.db' to not make any breaking changes. You can override the value using an environment variable.

brentvollebregt · 2020-07-13T09:52:05Z

docker/supervisor.d/supervisord.ini

+stdout_logfile=/dev/null
+stderr_logfile=/dev/null


I'm no expert on this, but wouldn't you want:

stderr_logfile=/dev/stderr stdout_logfile=/dev/stdout

What's your reasoning for not keeping the output?

UWSGI server creates a lot of uneccesary noise which does not give any value, hence we tune that down

brentvollebregt · 2020-07-13T09:54:38Z

README.md

@@ -96,6 +96,93 @@ from server import app as application

 > Alternatively these config values can be manually set in `config.py`.

+## Persistent SQLite Storage to S3 & Docker


Looks like some of the header numberings might be a bit off and it would make sense to put Persistent SQLite Storage to S3 & Docker under Docker maybe.

This file is getting pretty big, I might look at creating a /docs/ directory that we can relatively link to for more detailed documentation.

brentvollebregt · 2020-07-13T10:17:01Z

@brentvollebregt the S3 backup is optional in that scenario

you are not forcing anyone, if you don't specify the keys, backup would not do anything. If there is local db it would not overwrite from s3

there is a VOLUME option provided so you can just stick to it if that is your preference, check the README how to use it

I see now, in the README you have:

docker run --rm -ti --volumes-from hitcounter-data -p 80:80 -e AWS_PROFILE=wasabi -e AWS_ACCESS_KEY_ID="X" -e AWS_SECRET_ACCESS_KEY="Y" hitcounter

I was looking for something without AWS_PROFILE, AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY

* Simplified Dockerfile with nginx+uwsgi * Update docker info * Also removed default supervisord.ini file as this comes by default from the original image https://github.com/tiangolo/uwsgi-nginx-docker/blob/master/docker-images/supervisord-alpine.ini and we don't need to modify it as I originally thought

…e original image https://github.com/tiangolo/uwsgi-nginx-docker/blob/master/docker-images/supervisord-alpine.ini and we don't need to modify it as I originally thought

fluential · 2020-07-13T15:27:01Z

@brentvollebregt I've cleaned up the docker info which perhaps was not the most obvious, plus some other various cleanups -- have a look

brentvollebregt added 30 commits April 23, 2018 15:47

Initial commit

666eb02

Basic Layout

94f75d5

/count complete

71e8602

Fixed missing duplication for specified time

e5fa0e1

Cleaning and notes

ce2495b

Add expiration and svg

49144af

nocounts added

1471959

Clean database before interaction. Decided not to compress server.py …

36f3191

…code as is easier to read as-is

Put two return types in separate methods

fe86bdb

Readme, descriptions and tasks added

6479fa1

Add inspiration

34adbbc

Don't record a view if /nocount is requested

3d6508f

+ Development on index.html

Completed index.html

a3f1479

Calculate on page load for consistency

89c3be9

Site is now being hosted

7211dba

Add hit counter

7b59bc4

Disable caching

39fbc6d

Github caching is messing with me

4cf39ed

Update readme code blocks and make the cookie expire time a bit more …

4a9ac24

…generous

Add notice

0046324

Bug Fix

0f261af

- Remove converting a string to a string?

Add interface hosted from brentvollebregt.github.io

f9cafe9

Dynamic sizing for counts with a length above 5

4746e92

Ignore queries

679bca4

Ignore scheme and parse urls better

2eff533

Fix broken image

a978527

Additional fixes on readme

16a2bd3

Removed check URL count

10594de

+ Added HTTPS messages + code

Fixed a lot of the README and changed http to https

b9e5d17

Fix for "No 'Access-Control-Allow-Origin' header is present on the re…

6d806ed

…quested resource."

fluential added 7 commits July 11, 2020 08:19

Update info

f4b142a

Add database volume

2b2609c

Ensure dedicated db location

9f7e8ce

Enables local docker volume db store

Ensure optional buckt and db path

c8cec0f

Ensure optional bucket and db path

b3d99bb

Ensure optional bucket and db path

e17f7d5

Update

b1a8a4e

fluential added 3 commits July 11, 2020 08:49

Disable uwsgi logs

0e0584f

Not needed

Update

e267e76

Update

da40188

fluential mentioned this pull request Jul 11, 2020

Hosting on heroku #15

Open

feat: introduce url whitelist (#8)

ad29166

brentvollebregt requested changes Jul 13, 2020

View reviewed changes

fluential and others added 3 commits July 13, 2020 16:21

Merge branch 'master' into dockerize

9d93569

Removed default supervisord.ini file as this comes by default from th…

92c8af4

…e original image https://github.com/tiangolo/uwsgi-nginx-docker/blob/master/docker-images/supervisord-alpine.ini and we don't need to modify it as I originally thought

Typo fix

4d7ead1

fluential closed this Jul 14, 2020

fluential deleted the dockerize branch July 14, 2020 10:18

fluential restored the dockerize branch July 14, 2020 10:18

fluential deleted the dockerize branch July 14, 2020 10:18

fluential restored the dockerize branch July 14, 2020 10:23

fluential deleted the dockerize branch July 14, 2020 10:24

fluential restored the dockerize branch July 14, 2020 10:24

fluential mentioned this pull request Jul 14, 2020

Add dockerfile with ngxin+uwsgi, volumes and optional S3 storage #16

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 Persistent storage with Dockerfile nginx+uwsgi #13

S3 Persistent storage with Dockerfile nginx+uwsgi #13

fluential commented Jul 11, 2020 •

edited

fluential commented Jul 11, 2020

brentvollebregt commented Jul 12, 2020

fluential commented Jul 12, 2020 •

edited

brentvollebregt Jul 13, 2020

brentvollebregt Jul 13, 2020

fluential Jul 13, 2020

brentvollebregt Jul 13, 2020

brentvollebregt Jul 13, 2020

fluential Jul 13, 2020

brentvollebregt Jul 13, 2020

brentvollebregt commented Jul 13, 2020

fluential commented Jul 13, 2020 •

edited


		EXPOSE 8080
		RUN cd /usr/local/bin && wget https://raw.githubusercontent.com/fluential/docker-sqlite-to-s3/master/sqlite-to-s3.sh && chmod 755 sqlite*

		@@ -96,6 +96,93 @@ from server import app as application

		> Alternatively these config values can be manually set in `config.py`.

		## Persistent SQLite Storage to S3 & Docker

		stdout_logfile=/dev/null
		stderr_logfile=/dev/null

S3 Persistent storage with Dockerfile nginx+uwsgi #13

S3 Persistent storage with Dockerfile nginx+uwsgi #13

Conversation

fluential commented Jul 11, 2020 • edited

fluential commented Jul 11, 2020

brentvollebregt commented Jul 12, 2020

fluential commented Jul 12, 2020 • edited

brentvollebregt Jul 13, 2020

Choose a reason for hiding this comment

brentvollebregt Jul 13, 2020

Choose a reason for hiding this comment

fluential Jul 13, 2020

Choose a reason for hiding this comment

brentvollebregt Jul 13, 2020

Choose a reason for hiding this comment

brentvollebregt Jul 13, 2020

Choose a reason for hiding this comment

fluential Jul 13, 2020

Choose a reason for hiding this comment

brentvollebregt Jul 13, 2020

Choose a reason for hiding this comment

brentvollebregt commented Jul 13, 2020

fluential commented Jul 13, 2020 • edited

fluential commented Jul 11, 2020 •

edited

fluential commented Jul 12, 2020 •

edited

fluential commented Jul 13, 2020 •

edited