Put peqp in a separate volume called "static" #27

icyflame · 2017-10-27T20:27:19Z

I hope this is what you meant in your comment in #6 . The changes were pretty easy to make once I got the hang of it. 😄

This patch makes the mediawiki volume completely ephemeral
All static assets can be served from this subdomain: static.metakgp.org
Serving static assets from the same domain as the wiki turns out to be tricky, so I avoided it. It's just a DNS record and we can easily keep track of it.
The script at the end of this patch does the same thing as the restore mysql DB does. It is convenient to use:
- On a pc in the KGP network: wget -r 10.17.32.9
- Compress: tar czf peqp.tar.gz 10.17.32.9/
- Rsync to server: rsync -avzhe "ssh" peqp.tar.gz root@IP:~
- Extract at server: tar zxf peqp.tar.gz
- Move to metakgp-wiki and then call the script: ./script/restore-peqp-to-static.sh 10.17.32.9 (The first argument is the directory that consists the peqp folder alongwith all the PDFs)
I will add this to the runbook as soon as these two patches are merged in after testing it one more time.

amrav

Looks good, almost ready to merge - thanks!

amrav · 2017-10-28T00:38:51Z

nginx/Dockerfile

@@ -1,4 +1,5 @@
 FROM nginx:mainline
 COPY metakgp.org /etc/nginx/sites-enabled/


Can you rename these to wiki.metakgp.org and static.metakgp.org?

amrav · 2017-10-28T00:40:28Z

nginx/metakgp.org

@@ -27,7 +27,7 @@ server {
        autoindex off;

 	# Make site accessible from everywhere
-	server_name _;


Unfortunately I think this causes things like locahost to stop working, which we want for development. Can you verify whether this is the case, and see if there's a way around it?

Hey, I don't have docker installed on my computer. I did wget localhost on the server and that just gave me a 301 to the server URL. I will see if I have the space on my computer to set the whole thing up and check this.

amrav · 2017-10-28T00:42:44Z

nginx/metakgp.org

-	location /peqp {
-		try_files $uri $uri/ =404;
-	}
-
 	location ~ ^/google557cb96b33ddc6b5\.html$ {}

 	location /images {


For the mediawiki volume to be truly ephemeral, we also want to put image uploads into the static volume. You could take a look at whether it's possible to configure media uploads to a separate subdomain, but I think the simplest solution might be to symlink the images dir to a dir in the static volume. Feel free to defer this to a later patch though.

Yeah, I got it. Will defer to a later patch.

amrav · 2017-10-28T00:45:35Z

nginx/static

+
+	root /srv/static;
+	# index index.php index.html;
+        # autoindex off;


Does this turn off showing an index of all the files? I think we want that to be off, but what do you think?

Yeah, I think we should turn it off too. The default is off. I will keep this in the config to keep it explicit.

amrav · 2017-10-28T00:51:43Z

nginx/static

@@ -0,0 +1,60 @@
+# You may add here your


We can get rid of this boilerplate, I think.

amrav · 2017-10-28T00:53:20Z

docker-compose.override.yml

@@ -7,6 +7,7 @@ services:
  nginx:
    volumes:
      - mediawiki-volume:/srv/mediawiki
+      - static-volume:/srv/static


I think this belongs to the prod configuration - default and override should only contain config for development.

amrav · 2017-10-28T00:54:58Z

scripts/restore-peqp-to-static.sh

+
+source .env
+NGINX_CONTAINER=$(docker ps --format '{{ .Names }}' | grep nginx)
+docker cp $1/. $NGINX_CONTAINER:/srv/static


I have a slight preference against having scripts to do very simple things, like run docker cp. Perhaps this belongs to the runbook instead?

Hmm, We could put this in the runbook, this script just felt natural beside the mysql backup restore script. I think we should keep it.

icyflame · 2017-11-01T10:41:29Z

Rebased to latest master.

Remaining:

Check whether localhost works
Decide whether to have a separate script for restoring PEQP or put that in the runbook (I prefer having a script)

https://nginx.org/en/docs/http/ngx_http_autoindex_module.html#autoindex

icyflame · 2017-11-06T15:01:27Z

@amrav Localhost still works. I just setup a new server and tried it and it works. I have rebased to latest master and used this particular build to set this up. Please review and approve! 🙂

icyflame · 2017-11-12T14:40:30Z

Hey @amrav, I have solved the issues now. Can you please review and merge? I think all the requirements have been met.

amrav · 2017-11-13T17:46:17Z

Lgtm if it works, but do you understand why nginx serves the correct domain for localhost? Why isn't it defaulting to static.metakgp.org, for instance?

icyflame · 2017-11-14T08:08:43Z

@amrav Okay, the mechanism was easy to understand. It's well documented here

In particular:

A verbose log of the request:

So, the Host: localhost doesn't match anything configured and it just routes the request to the default_server which we have configured to be the wiki server.

I hope that clears it up. This was a good thing to find out! 😄 Okay to merge this patch now?

amrav · 2017-11-14T10:34:57Z

Sounds good, thanks for clarifying!

…

On Tue, 14 Nov 2017 at 08:08, Siddharth Kannan ***@***.***> wrote: @amrav <https://github.com/amrav> Okay, the mechanism was easy to understand. It's well documented here <https://nginx.org/en/docs/http/request_processing.html> In particular: [image: image] <https://user-images.githubusercontent.com/3668034/32769172-c9f135d0-c940-11e7-9f53-b2544764cf93.png> A verbose log of the request: [image: image] <https://user-images.githubusercontent.com/3668034/32769184-d8cd06f6-c940-11e7-9e15-9acb358769c8.png> So, the Host: localhost doesn't match anything configured and it just routes the request to the default_server which we have configured to be the wiki server. I hope that clears it up. This was a good thing to find out! 😄 Okay to merge this patch now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#27 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABIZ_7ts5HjS8vnZfVr311KSVFobCV7Gks5s2UqMgaJpZM4QJk4b> .

icyflame requested a review from amrav October 27, 2017 20:27

amrav requested changes Oct 28, 2017

View reviewed changes

icyflame force-pushed the icyflame/put-peqp-in-static branch from 96d1275 to efb30e7 Compare November 1, 2017 10:39

icyflame added 9 commits November 6, 2017 19:36

create a new static volume for peqp

aa5ed6c

remove default_server from nginx/static

73256ed

add a script to restore peqp from backup

a6d8da9

serve everything from static folder for this server_name

39fa161

serve wiki only for one server_name

788e4c3

rename conf files appropriately

d4f7e5e

move definition of static-volume to prod override

4712a97

remove boilerplate from nginx conf files

aae29aa

Turn autoindex off

392baec

https://nginx.org/en/docs/http/ngx_http_autoindex_module.html#autoindex

icyflame force-pushed the icyflame/put-peqp-in-static branch from efb30e7 to 392baec Compare November 6, 2017 14:06

amrav approved these changes Nov 13, 2017

View reviewed changes

icyflame merged commit ae89734 into metakgp:master Nov 14, 2017

icyflame mentioned this pull request Nov 14, 2017

Add peqp to monthly backups #6

Closed

icyflame deleted the icyflame/put-peqp-in-static branch November 25, 2017 06:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Put peqp in a separate volume called "static" #27

Put peqp in a separate volume called "static" #27

icyflame commented Oct 27, 2017

amrav left a comment

amrav Oct 28, 2017

amrav Oct 28, 2017

icyflame Oct 29, 2017

amrav Oct 28, 2017

icyflame Oct 29, 2017

amrav Oct 28, 2017

icyflame Oct 29, 2017

amrav Oct 28, 2017

amrav Oct 28, 2017

amrav Oct 28, 2017

icyflame Oct 29, 2017

icyflame commented Nov 1, 2017 •

edited by ghostwriternr

Loading

icyflame commented Nov 6, 2017

icyflame commented Nov 12, 2017

amrav commented Nov 13, 2017

icyflame commented Nov 14, 2017

amrav commented Nov 14, 2017 via email

		@@ -1,4 +1,5 @@
		FROM nginx:mainline
		COPY metakgp.org /etc/nginx/sites-enabled/

Put peqp in a separate volume called "static" #27

Put peqp in a separate volume called "static" #27

Conversation

icyflame commented Oct 27, 2017

amrav left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

icyflame commented Nov 1, 2017 • edited by ghostwriternr Loading

icyflame commented Nov 6, 2017

icyflame commented Nov 12, 2017

amrav commented Nov 13, 2017

icyflame commented Nov 14, 2017

amrav commented Nov 14, 2017 via email

icyflame commented Nov 1, 2017 •

edited by ghostwriternr

Loading