Yunohost package #2

Thatoo · 2024-01-02T12:00:32Z

Thatoo
Jan 2, 2024
Maintainer

I've made good progress on my Dokos bash install script.
You can find it here : https://github.com/Thatoo/Dokos_ynh/tree/Thatoo-bash-script-install/bash-script-install

In order to make it a yunohost package, I have two questions :

Dokos overwrite /etc/nginx/nginx.conf with a specific nginx.conf file that you can find at the same url
Is it really a need? I'm not able to judge the differences between this specific ngix.conf file and a standard yunohost nginx config file.

Is there a way in Yunohost packaging to deal with this situation?

Because the command
bench new-site $site_name --db-name $db_name --db-password $db_pwd --no-setup-db --admin-password $admin_pwd
check, before anything, if running the command SHOW VARIABLES in mysql, will give back these two values :

	"character_set_server": "utf8mb4",
	"collation_server": "utf8mb4_unicode_ci",

Then in the script, I need to ad this

sudo bash -c 'cat << EOF >> /etc/mysql/my.cnf
[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

[mysql]
default-character-set = utf8mb4
EOF'

sudo service mysql restart

before running the command
bench new-site $site_name --db-name $db_name --db-password $db_pwd --no-setup-db --admin-password $admin_pwd

Is there a way in Yunohost packaging to deal with this situation?

Thatoo · 2024-01-02T12:35:53Z

Thatoo
Jan 2, 2024
Maintainer Author

Oh, and another thing that this script doesn't manage (whereas the command bench setup production does, this command is removed from the script to be yunohost compliant) is fail2ban.
The Yunohost script should take care of it too.

bench setup production was doing the following :

Install fail2ban
Enable fail2ban
Create jail.d
Setup filters
setup jails
restart fail2ban

I added the following files in the "bash-script-install" folder to help with that :

/etc/fail2ban/jail.d/nginx-proxy.conf
/etc/fail2ban/filter.d/nginx-proxy.conf

With them and solving my two questions aove, I believe we could make a Yunohost package now.

0 replies

Thatoo · 2024-01-02T16:49:48Z

Thatoo
Jan 2, 2024
Maintainer Author

This script works also fro Debian 12 but it needs some adjustment because it's not python3.10 but python3.11.
I have tested it and it works so it will be ready for yunohost 12 too then if we manage to make a package.

0 replies

Thatoo · 2024-01-02T16:53:46Z

Thatoo
Jan 2, 2024
Maintainer Author

If I check the differences between the Yunohost /etc/nginx/nginx.conf and the nginx.conf file provided by Dokos, here are the additional lines in the dokos nginx.conf file :

worker_rlimit_nofile 65535;
events {
    worker_connections 1024;
    multi_accept on;
}


http {

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';


    tcp_nodelay on;
    server_tokens off;


    gzip_disable "msie6";
    gzip_http_version 1.1;
    gzip_comp_level 5;
    gzip_min_length 256;
    gzip_proxied any;
    gzip_vary on;
    gzip_types
        application/atom+xml
        application/javascript
        application/json
        application/rss+xml
        application/vnd.ms-fontobject
        application/x-font-ttf
        application/font-woff
        application/x-web-app-manifest+json
        application/xhtml+xml
        application/xml
        font/opentype
        image/svg+xml
        image/x-icon
        text/css
        text/plain
        text/x-component
        ;

    server_names_hash_max_size 4096;

    open_file_cache max=65000 inactive=1m;
    open_file_cache_valid 5s;
    open_file_cache_min_uses 1;
    open_file_cache_errors on;

    ssl_protocols  SSLv3 TLSv1;
    ssl_ciphers     ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH:!AESGCM;

    client_max_body_size 50m;
    large_client_header_buffers 4 32k;

    proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=web-cache:8m max_size=1000m inactive=600m;

}

How could I merge these lines into the /tec/nginx/conf.d/dokos.conf given fil (which for now is integrated into the script as EOF ?

upstream dokos-bench-folder-frappe {
	server 127.0.0.1:8000 fail_timeout=0;
}

upstream dokos-bench-folder-socketio-server {
	server 127.0.0.1:9000 fail_timeout=0;
}



# setup maps


# server blocks





server {
	
	listen 80;
	listen [::]:80;
	

	server_name dokos.local;

	root /home/thatoo/dokos-bench-folder/sites;

	

	proxy_buffer_size 128k;
	proxy_buffers 4 256k;
	proxy_busy_buffers_size 256k;

	

	add_header X-Frame-Options "SAMEORIGIN";
	add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
	add_header X-Content-Type-Options nosniff;
	add_header X-XSS-Protection "1; mode=block";
	add_header Referrer-Policy "same-origin, strict-origin-when-cross-origin";

	location /assets {
		try_files $uri =404;
	}

	location ~ ^/protected/(.*) {
		internal;
		try_files /dokos.local/$1 =404;
	}

	location /socket.io {
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header X-Frappe-Site-Name dokos.local;
		proxy_set_header Origin $scheme://$http_host;
		proxy_set_header Host $host;

		proxy_pass http://dokos-bench-folder-socketio-server;
	}

	location ^~ /.well-known/acme-challenge/ {
		default_type "text/plain";
		root /var/www/letsencrypt;
	}

	location = /.well-known/acme-challenge/ {
		return 404;
	}

	location / {

		rewrite ^(.+)/$ $1 permanent;
  		rewrite ^(.+)/index\.html$ $1 permanent;
  		rewrite ^(.+)\.html$ $1 permanent;

		location ~* ^/files/.*.(htm|html|svg|xml) {
			add_header Content-disposition "attachment";
			try_files /dokos.local/public/$uri @webserver;
		}

		try_files /dokos.local/public/$uri @webserver;
	}

	location @webserver {
		proxy_http_version 1.1;
		proxy_set_header X-Forwarded-For $remote_addr;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Frappe-Site-Name dokos.local;
		proxy_set_header Host $host;
		proxy_set_header X-Use-X-Accel-Redirect True;
		proxy_read_timeout 120;
		proxy_redirect off;

		proxy_pass  http://dokos-bench-folder-frappe;
	}

	# error pages
	error_page 502 /502.html;
	location /502.html {
		root /usr/local/lib/python3.10/dist-packages/bench/config/templates;
		internal;
	}

	

	# optimizations
	sendfile on;
	keepalive_timeout 15;
	client_max_body_size 50m;
	client_body_buffer_size 16K;
	client_header_buffer_size 1k;

	# enable gzip compresion
	# based on https://mattstauffer.co/blog/enabling-gzip-on-nginx-servers-including-laravel-forge
	gzip on;
	gzip_http_version 1.1;
	gzip_comp_level 5;
	gzip_min_length 256;
	gzip_proxied any;
	gzip_vary on;
	gzip_types
		application/atom+xml
		application/javascript
		application/json
		application/rss+xml
		application/vnd.ms-fontobject
		application/x-font-ttf
		application/font-woff
		application/x-web-app-manifest+json
		application/xhtml+xml
		application/xml
		font/opentype
		image/svg+xml
		image/x-icon
		text/css
		text/plain
		text/x-component
		;
		# text/html is always compressed by HttpGzipModule
}

0 replies

Thatoo · 2024-01-02T17:13:15Z

Thatoo
Jan 2, 2024
Maintainer Author

I guess I can add everything which is within http to the conf.d/dokos.conf file but what about these lines ?

worker_rlimit_nofile 65535;
events {
    worker_connections 1024;
    multi_accept on;
}

1 reply

cogk Jan 2, 2024

I don't know enough about Yunohost to help here. I think these lines are not necessary if the base config of Nginx/Yunohost already specifies these settings (related to the handling of concurrent requests).

Thatoo · 2024-01-02T20:38:18Z

Thatoo
Jan 2, 2024
Maintainer Author

I found where are coming the two files.
The file supposed to replace /etc/nginx/nginx.conf comes from https://gitlab.com/dokos/docli/-/blob/master/bench/playbooks/roles/nginx/templates/nginx.conf.j2?ref_type=heads
The file supposed to be added to /etc/nginx/conf.d/dokos.conf comes from : https://gitlab.com/dokos/docli/-/blob/master/bench/config/templates/nginx.conf?ref_type=heads

For the reccord, here comes the supervisor.conf file : https://gitlab.com/dokos/docli/-/blob/master/bench/config/templates/supervisor.conf?ref_type=heads

0 replies

Thatoo · 2024-01-03T00:03:22Z

Thatoo
Jan 3, 2024
Maintainer Author

I end up to this situation that I hope is satisfying.
No more

sudo mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.old
sudo cp $script_folder/nginx.conf /etc/nginx/nginx.conf

Instead the following /etc/nginx/conf.d/dokos.conf file :


upstream dokos-bench-folder-frappe {
	server 127.0.0.1:8000 fail_timeout=0;
}

upstream dokos-bench-folder-socketio-server {
	server 127.0.0.1:9000 fail_timeout=0;
}

server {
	
	listen 80;
	listen [::]:80;

	server_name dokos.local;

	root /home/thatoo/dokos-bench-folder/sites;

	proxy_buffer_size 128k;
	proxy_buffers 4 256k;
	proxy_busy_buffers_size 256k;

	add_header X-Frame-Options "SAMEORIGIN";
	add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
	add_header X-Content-Type-Options nosniff;
	add_header X-XSS-Protection "1; mode=block";
	add_header Referrer-Policy "same-origin, strict-origin-when-cross-origin";

	location /assets {
		try_files $uri =404;
	}

	location ~ ^/protected/(.*) {
		internal;
		try_files /dokos.local/$1 =404;
	}

	location /socket.io {
		proxy_http_version 1.1;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";
		proxy_set_header X-Frappe-Site-Name dokos.local;
		proxy_set_header Origin $scheme://$http_host;
		proxy_set_header Host dokos.local;

		proxy_pass http://dokos-bench-folder-socketio-server;
	}

	location ^~ /.well-known/acme-challenge/ {
		default_type "text/plain";
		root /var/www/letsencrypt;
	}

	location = /.well-known/acme-challenge/ {
		return 404;
	}

	location / {

		rewrite ^(.+)/$ $1 permanent;
  		rewrite ^(.+)/index\.html$ $1 permanent;
  		rewrite ^(.+)\.html$ $1 permanent;

		location ~* ^/files/.*.(htm|html|svg|xml) {
			add_header Content-disposition "attachment";
			try_files /dokos.local/public/$uri @webserver;
		}

		try_files /dokos.local/public/$uri @webserver;
	}

	location @webserver {
		proxy_http_version 1.1;
		proxy_set_header X-Forwarded-For $remote_addr;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header X-Frappe-Site-Name dokos.local;
		proxy_set_header Host dokos.local;
		proxy_set_header X-Use-X-Accel-Redirect True;
		proxy_read_timeout 120;
		proxy_redirect off;

		proxy_pass  http://dokos-bench-folder-frappe;
	}

	# error pages
	error_page 502 /502.html;
	location /502.html {
		root /usr/local/lib/python3.10/dist-packages/bench/config/templates;
		internal;
	}

	# optimizations
	sendfile on;
	keepalive_timeout 15;
	client_max_body_size 50m;
	client_body_buffer_size 16K;
	client_header_buffer_size 1k;
	large_client_header_buffers 4 32k;

	# enable gzip compresion
	gzip on;
	gzip_http_version 1.1;
	gzip_comp_level 5;
	gzip_min_length 256;
	gzip_proxied any;
	gzip_vary on;
	gzip_types
		application/atom+xml
		application/javascript
		application/json
		application/rss+xml
		application/vnd.ms-fontobject
		application/x-font-ttf
		application/font-woff
		application/x-web-app-manifest+json
		application/xhtml+xml
		application/xml
		font/opentype
		image/svg+xml
		image/x-icon
		text/css
		text/plain
		text/x-component
		;

	tcp_nodelay on;
	server_tokens off;

	open_file_cache max=65000 inactive=1m;
	open_file_cache_valid 5s;
	open_file_cache_min_uses 1;
	open_file_cache_errors on;
}

And No more

sudo bash -c 'cat << EOF >> /etc/mysql/my.cnf
[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci

[mysql]
default-character-set = utf8mb4
EOF'

sudo service mysql restart

I tricked the command bench new-site this way :

#to trick the command bench new-site which check value of character_set_server and collation_server even though we created a database setting character and collation for the database itself
character_set_server=$(sudo mysql -u root -ss -N -e "SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_VARIABLES WHERE  variable_name ='character_set_server';")
collation_server=$(sudo mysql -u root -ss -N -e "SELECT VARIABLE_VALUE FROM information_schema.GLOBAL_VARIABLES WHERE  variable_name ='collation_server';")
sudo mysql -u root -e "SET GLOBAL character_set_server = 'utf8mb4'";
sudo mysql -u root -e "SET GLOBAL collation_server = 'utf8mb4_unicode_ci'";

bench new-site $site_name --db-name $db_name --db-password $db_pwd --no-setup-db --admin-password $admin_pwd

# putting back the original value of character_set_server and collation_server
sudo mysql -u root -e "SET GLOBAL character_set_server = '$character_set_server'";
sudo mysql -u root -e "SET GLOBAL collation_server = '$collation_server'";

It's not very nice but it works. I hope it will work also for a Yunohost package.

0 replies

Thatoo · 2024-01-03T12:27:14Z

Thatoo
Jan 3, 2024
Maintainer Author

[Update] Instead of tricking bench new-site modifying mysql GLOBAL VARIABLES which could impact other part of Yunohost so it wasn't a satisfying solution, let's modify Dokos python file and let it check character_set_database and collation_database insteadf of character_set_server and collation_server

So the script becomes

#Let's make the command bench new-site check value of character_set_database and collation_database instead of character_set_server and collation_server
sed -i 's/"character_set_server": "utf8mb4",/"character_set_database": "utf8mb4",/g' $benchdir/apps/frappe/frappe/database/mariadb/setup_db.py
sed -i 's/"collation_server": "utf8mb4_unicode_ci",/"collation_database": "utf8mb4_unicode_ci",/g' $benchdir/apps/frappe/frappe/database/mariadb/setup_db.py

bench new-site $site_name --db-name $db_name --db-password $db_pwd --no-setup-db --admin-password $admin_pwd

which is much better and make me more confident to start on a Yunohost package.

0 replies

Thatoo · 2024-01-06T21:22:23Z

Thatoo
Jan 6, 2024
Maintainer Author

In order to avoid these two sed command line, I have made an upstream PR : frappe/frappe#24150
We'll see if it gets accepted.

0 replies

Thatoo · 2024-04-10T14:43:33Z

Thatoo
Apr 10, 2024
Maintainer Author

Ce PR n'est pas passé mais un autre oui et a été intégré au code de frappe puis de Dodock.
Il existe donc maintenant un script d'installation automatique pour debian 12 et ubuntu 22.04 et supérieur sans sed ni autre "ruse" :
https://github.com/Thatoo/Dokos_ynh/blob/Thatoo-bash-script-install/bash-script-install/dokos_install.sh

Il me semble que ce script peut maintenant servir de base à la création d'un paquet Yunohost 12.

1 reply

cogk Apr 10, 2024

Super ! 👏

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Yunohost package #2

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 9 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Yunohost package #2

Thatoo Jan 2, 2024 Maintainer

Replies: 9 comments · 2 replies

Thatoo Jan 2, 2024 Maintainer Author

Thatoo Jan 2, 2024 Maintainer Author

Thatoo Jan 2, 2024 Maintainer Author

Thatoo Jan 2, 2024 Maintainer Author

cogk Jan 2, 2024

Thatoo Jan 2, 2024 Maintainer Author

Thatoo Jan 3, 2024 Maintainer Author

Thatoo Jan 3, 2024 Maintainer Author

Thatoo Jan 6, 2024 Maintainer Author

Thatoo Apr 10, 2024 Maintainer Author

cogk Apr 10, 2024

Thatoo
Jan 2, 2024
Maintainer

Replies: 9 comments 2 replies

Thatoo
Jan 2, 2024
Maintainer Author

Thatoo
Jan 2, 2024
Maintainer Author

Thatoo
Jan 2, 2024
Maintainer Author

Thatoo
Jan 2, 2024
Maintainer Author

Thatoo
Jan 2, 2024
Maintainer Author

Thatoo
Jan 3, 2024
Maintainer Author

Thatoo
Jan 3, 2024
Maintainer Author

Thatoo
Jan 6, 2024
Maintainer Author

Thatoo
Apr 10, 2024
Maintainer Author