Dockerfile #450

Open
wants to merge 1 commit into
from

Projects

None yet

5 participants

@galexrt
Contributor
galexrt commented Jan 29, 2016 edited

@timabbott Here you go, a working Dockerfile. Any improvements I could add?

This adds a working Dockerfile and my create-admin-user.sh script.

I included the docker-compose.yml because the database server needs to be my galexrt/zulip-postgresql-tsearchextras (because other postgresql images don't have the tsearch extension installed). You would just need to change the image names (example zulip/zulip:1.3.10).

If there is anything that can improve this Pull Request, please let me know.


Original commit messages:

This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

@smarx I signed the CLA.

@smarx
smarx commented Jan 29, 2016

Automated message from Dropbox CLA bot

@Galexrt, thanks for the pull request! It looks like you haven't yet signed the Dropbox CLA. Please sign it here and update the thread so we can consider merging your code.

@galexrt
Contributor
galexrt commented Jan 29, 2016

@smarx I signed it (twice now)...

@smarx
smarx commented Jan 29, 2016

@Galexrt Actually, you've signed it four times. That was the problem... the script used to fail if you had signed more than once (it being unclear which response to use). Now it checks only the most recent signature, so I believe it will pass the next time you try.

@timabbott
Member

@Galexrt awesome! I'm travelling this weekend but will take a look next week.

@timabbott timabbott commented on an outdated diff Feb 4, 2016
+ rm -rf /root/zulip/puppet/ && \
+ cp -fa /root/zulip/zproject/local_settings_template.py /etc/zulip/settings.py && \
+ ln -nsf /etc/zulip/settings.py /root/zulip/zproject/local_settings.py && \
+ ZULIP_DEPLOY_PATH="$(/root/zulip/zulip_tools.py make_deploy_path)" && \
+ mv /root/zulip "$ZULIP_DEPLOY_PATH" && \
+ ln -nsf "/home/zulip/deployments/next" /root/zulip && \
+ ln -nsf "$ZULIP_DEPLOY_PATH" "/home/zulip/deployments/next" && \
+ ln -nsf "$ZULIP_DEPLOY_PATH" "/home/zulip/deployments/current" && \
+ ln -nsf /etc/zulip/settings.py "$ZULIP_DEPLOY_PATH/zproject/local_settings.py" && \
+ wget -q https://www.zulip.com/dist/releases/zulip-server-latest.tar.gz -O /tmp/zulip-server.tar.gz && \
+ tar xfz /tmp/zulip-server.tar.gz -C "/home/zulip/prod-static" --strip-components=3 --wildcards */prod-static/serve && \
+ rm -rf /tmp/zulip-server.tar.gz && \
+ chown -R zulip:zulip /home/zulip /var/log/zulip /etc/zulip/settings.py && \
+ apt-get -qq autoremove --purge -y && \
+ apt-get -qq clean && \
+ rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
@timabbott
timabbott Feb 4, 2016 Member

It seems like this block of code could be a thin wrapper around scripts/setup/install, possible with an option or two added similar to $APT_OPTIONS, right? Things would be a lot cleaner if we didn't have to maintain a second copy of all this setup code.

@timabbott timabbott commented on an outdated diff Feb 4, 2016
docker-compose.yml
+memcached:
+ image: "quay.io/sameersbn/memcached:latest"
+ restart: always
+rabbitmq:
+ image: "rabbitmq:3.5.5"
+ hostname: zulip-rabbit
+ restart: always
+ environment:
+ RABBITMQ_DEFAULT_USER: "zulip"
+ RABBITMQ_DEFAULT_PASS: "zulip"
+redis:
+ image: "quay.io/sameersbn/redis:latest"
+ volumes:
+ - "/opt/docker/zulip/redis:/var/lib/redis:rw"
+zulip:
+ image: "quay.io/galexrt/docker-zulip:docker"
@timabbott
timabbott Feb 4, 2016 Member

Is an image needed in this version?

@timabbott timabbott commented on the diff Feb 4, 2016
puppet/zulip/files/supervisor/conf.d/nginx.conf
@@ -0,0 +1,5 @@
+[program:nginx]
@timabbott
timabbott Feb 4, 2016 Member

Can you add a comment for this one similar to the cron.conf comment?

@galexrt
galexrt Feb 4, 2016 Contributor

Comment added.

@timabbott
timabbott Mar 19, 2016 Member

Great, thanks!

@timabbott timabbott commented on an outdated diff Feb 4, 2016
scripts/create-admin-user.sh
+ read -s -p ZULIP_USER_DOMAIN
+done
+while [[ -z "$ZULIP_USER_PASS" ]]; do
+ echo "Please enter the Zulip User Full name:"
+ read -s -p ZULIP_USER_FULLNAME
+done
+while [[ -z "$ZULIP_USER_PASS" ]]; do
+ echo "Please enter the Zulip User Email:"
+ read -s -p ZULIP_USER_EMAIL
+done
+while [[ -z "$ZULIP_USER_PASS" ]]; do
+ echo "Please enter the Zulip User Password:"
+ read -s -p ZULIP_USER_PASS
+done
+
+# Doing everything in python, even I never coded in python #YOLO - Alexander T.
@timabbott
timabbott Feb 4, 2016 Member

Hehe nice first python program :)

@timabbott timabbott commented on an outdated diff Feb 4, 2016
scripts/create-admin-user.sh
+
+try:
+ realm = get_realm('$ZULIP_USER_DOMAIN')
+except Realm.DoesNotExist:
+ raise CommandError("Realm/Domain does not exist.")
+
+try:
+ create_user('$ZULIP_USER_EMAIL', '$ZULIP_USER_PASS', realm, '$ZULIP_USER_FULLNAME', email_to_username('$ZULIP_USER_EMAIL'))
+except:
+ pass
+
+User = get_user_profile_by_email(email='$ZULIP_USER_EMAIL')
+do_change_is_admin(User, True, 'administer')
+User.save()
+quit()
+EOF
@timabbott
timabbott Feb 4, 2016 Member

I don't think you need to write a separate script for this; you should be able to do it with:

./manage.py create_user <args> followed by manage.py knight -f user@example.com

The only reason that wouldn't do what you want is manage.py create_user (implemented in zerver/management/commands/create_user.py) doesn't support passing a password argument; can you instead just add such an option? I'd happily take that as a separate PR which we can quickly merge; no reason to bundle it in here.

Then this code can be just prompting the user and then running those 2 management commands.

@timabbott timabbott commented on the diff Feb 4, 2016
puppet/zulip/manifests/dockervoyager.pp
@@ -0,0 +1,32 @@
+class zulip::dockervoyager {
@timabbott
timabbott Feb 4, 2016 Member

Can you add a brief comment at the top explaining a bit more about what this is?

@galexrt
galexrt Feb 4, 2016 Contributor

I added a comment to the file. Is this okay?

@timabbott timabbott commented on an outdated diff Feb 4, 2016
@@ -27,6 +27,11 @@ ongoing on adding support for additional platforms. The installation
process is documented in https://zulip.org/server.html and in more
detail in [README.prod.md](README.prod.md).
+Running Zulip in Docker
+===========================
+
+Zulip in Docker. The installation process is documented in [README.docker.md](README.docker.md).
+
@timabbott
timabbott Feb 4, 2016 Member

Can you add an "(experimental)" note here? And also move this doc to be a subsection of README.prod.md instead of in the main README.md, since that's where we document installing Zulip for production?

@timabbott timabbott commented on an outdated diff Feb 4, 2016
README.docker.md
@@ -0,0 +1,74 @@
+Zulip
+=====
+
+Zulip is a powerful, open source group chat application. Written in
+Python and using the Django framework, Zulip supports both private
+messaging and group chats via conversation streams.
+
+Zulip also supports fast search, drag-and-drop file uploads, image
+previews, group private messages, audible notifications,
+missed-message emails, desktop apps, and much more.
+
+Further information on the Zulip project and its features can be found
+at https://www.zulip.org.
+
+Using Zulip with docker
@timabbott
timabbott Feb 4, 2016 Member

Add a "in production" and "(experimental)" in this heading?

@timabbott timabbott commented on an outdated diff Feb 4, 2016
README.docker.md
+openssl genrsa -out zulip.key 2048
+```
+
+**STEP 2**: Create the certificate signing request (CSR)
+
+```bash
+openssl req -new -key zulip.key -out zulip.csr
+```
+
+**STEP 3**: Sign the certificate using the private key and CSR
+
+```bash
+openssl x509 -req -days 3650 -in zulip.csr -signkey zulip.key -out zulip.crt
+```
+
+Congratulations! You have now generated the self-signed SSL certificate, it is valid for 10 years.
@timabbott
timabbott Feb 4, 2016 Member

This section duplicates the SSL part of README.prod.md; can you pull these out into a separate doc (README.ssl.md?) and link to it from both places?

@timabbott timabbott and 1 other commented on an outdated diff Feb 4, 2016
docker-entrypoint.sh
+ echo "(Re)creating database structure ..."
+ psql -h "$DB_HOST" -U "$DB_USER" <<EOF || :
+CREATE USER $DB_USER;
+ALTER ROLE $DB_USER SET search_path TO zulip,public;
+CREATE DATABASE $DB_NAME OWNER=$DB_USER;
+EOF
+ psql -h "$DB_HOST" -U "$DB_USER" <<EOF || :
+CREATE SCHEMA $DB_SCHEMA AUTHORIZATION $DB_USER;
+CREATE EXTENSION IF NOT EXISTS tsearch_extras SCHEMA zulip;
+EOF
+ unset PGPASSWORD
+ echo "Database structure (re)created."
+ else
+ echo "No database information given. Skipping ..."
+ fi
+}
@timabbott
timabbott Feb 4, 2016 Member

This section duplicates some somewhat tricky code from scripts/setup/postgres-init-db -- would it be possible to have this use that script (potentially with an option added to the script)? I'd prefer to not be maintaining more copies of that database setup code if we can avoid it.

@galexrt
galexrt Feb 4, 2016 Contributor

Like in the scripts/lib/install, I'll have to add some code there, but that shouldn't be a problem.

@timabbott
Member

Cool, thanks for refactoring this, @Galexrt ! This is looking a ton better. I posted a bunch of comments on how to further simplify the amount of code that this involves maintaining two copies of; I think with those things addressed this'll be fairly clean and reasonable to merge.

This does definitely remind me that we should probably try to get the tsearch_extras extension into postgres upstream, since maintaining a postgres extension module is kinda a pain.

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Feb 4, 2016
@galexrt galexrt Removed the create-admin-user.sh script (see zulip#450 (comment)) 5795d0e
@galexrt
Contributor
galexrt commented Feb 4, 2016 edited

@timabbott I had to change the scripts/lib/install a bit, so the Docker installation is possible without wrapping the code around (see your comment #450 (comment)).

@galexrt
Contributor
galexrt commented Feb 4, 2016

Adding the tsearch_extras to postgres upstream would be good, because it's a pain to maintain the special needed image Galexrt/docker-zulip-postgresql-tsearchextras. The image Galexrt/docker-zulip-postgresql-tsearchextras is needed because of the tsearch_extras, but it's not that much trouble.

BTW I'll rebase when everything is fixed and okay.

@galexrt
Contributor
galexrt commented Feb 4, 2016

@timabbott What package do I need to install to generate the prod-static files?
(I currently get the prod-static files through, downloading latest zulip version and extracting prod-static files)

@timabbott
Member

They're generated by tools/update-prod-static.

That said, it seems like by default the Docker-in-production setup should pull from a release tarball, not Git, since that's likely to be more stable for a production environment. It'd be great to have an option for it to deploy from Git though.

@galexrt
Contributor
galexrt commented Feb 4, 2016

@timabbott I added the Dockerfile-prod. The Dockerfile-prod uses the latest release tarball as you want, but building the Docker image will first work when the changes I made to the scripts are in the release tarball.

Dockerfile Overview:

  • Dockerfile uses the files from the directory ("branch" building) (good for testing?)
  • Dockerfile-prod uses latest release tarball
  • Dockerfile-dev is the old Dockerfile

Are the multiple Dockerfiles for different "builds" okay?

@galexrt
Contributor
galexrt commented Feb 6, 2016

@timabbott I get the following errors when running the update-prod-static tool.

Anything I can do against this error? Or shall I use the latest prod-static files even for the per branch dev image?

zulip_1     | npm WARN EPACKAGEJSON zulip@0.0.0 No description
zulip_1     | npm ERR! Linux 4.1.16-1-MANJARO
zulip_1     | npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install"
zulip_1     | npm ERR! node v5.5.0
zulip_1     | npm ERR! npm  v3.3.12
zulip_1     | npm ERR! code ELIFECYCLE
zulip_1     | 
zulip_1     | npm ERR! contextify@0.1.15 install: `node-gyp rebuild`
zulip_1     | npm ERR! Exit status 1
zulip_1     | npm ERR! 
zulip_1     | npm ERR! Failed at the contextify@0.1.15 install script 'node-gyp rebuild'.
zulip_1     | npm ERR! Make sure you have the latest version of node.js and npm installed.
zulip_1     | npm ERR! If you do, this is most likely a problem with the contextify package,
zulip_1     | npm ERR! not with npm itself.
zulip_1     | npm ERR! Tell the author that this fails on your system:
zulip_1     | npm ERR!     node-gyp rebuild
zulip_1     | npm ERR! You can get their info via:
zulip_1     | npm ERR!     npm owner ls contextify
zulip_1     | npm ERR! There is likely additional logging output above.
zulip_1     | 
zulip_1     | npm ERR! Please include the following file with any support request:
zulip_1     | npm ERR!     /home/zulip/deployments/2016-02-05-14-43-28/npm-debug.log
zulip_1     | Traceback (most recent call last):
zulip_1     |   File "/home/zulip/deployments/current/tools/update-prod-static", line 29, in <module>
zulip_1     |     subprocess.check_call(['npm', 'install'], stdout=fp, stderr=fp);
zulip_1     |   File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
zulip_1     |     raise CalledProcessError(retcode, cmd)
zulip_1     | subprocess.CalledProcessError: Command '['npm', 'install']' returned non-zero exit status 1
@timabbott
Member

I think the overall issue here is that there are some dependencies needed to build the static assets that we don't install in the production puppet files (since they aren't needed there). These include:

  • closure-compiler (currently doesn't show up in puppet at all, which I should probably fix)
  • nodejs
  • npm
  • yui-compressor
  • libfreetype6-dev
  • (possibly some more of the libraries from the list in provision.py? Those are the ones I'm confident of)
    I think probably the right fix for this is to add a new puppet module, e.g. puppet/zulip/manifests/static_build.pp that installs the dependencies needed to run the static asset pipeline.

That error in particular seems like it could be e.g. a permissions problem (that's what I see when i google ELIFECYCLE); I'd check the npm-debug.log to try to figure out what's up...

@galexrt
Contributor
galexrt commented Feb 8, 2016

@timabbott I have removed the npm build from the Dockerfile's and switched to using the Zulip tarball.

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Feb 8, 2016
@galexrt galexrt Part 1 of X changes from the comments
Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder
613145c
@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Feb 8, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder
5d9965f
@galexrt
Contributor
galexrt commented Feb 8, 2016

I just tested the image, with run docker-compose up, configured all configs it and restarted the containers. Et voilà a working Zulip instance. :)

The only thing that wasn't properly working, are the emoticons. They appear in the auto completion but not when the message is sent, but I'll look into this now.

@galexrt
Contributor
galexrt commented Feb 16, 2016

@timabbott Any news about merging this?

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Feb 17, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder

Should fix the Dockerfile-prod problems

Fix documentation around iOS not supporting a custom server.

Fix performance issues with user presence list in large realms.

Whenever a user became active, this triggers an immediate presence
update event (to show that user as active).  The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user.  This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.

[Commit message expanded with more details by Tim Abbott]

Upgrade image to latest package versions before starting
378453a
@galexrt
Contributor
galexrt commented Feb 17, 2016

The Dockerfile-prod will only work when there is a new release with the modified scripts and the added dockervoyager.pp in it .

@gvt
gvt commented Feb 26, 2016

+1 for these changes. would like to see this released so that docker-zulip can use it and fix the error.

@timabbott timabbott and 1 other commented on an outdated diff Mar 19, 2016
scripts/lib/install
deploy_path=$(/root/zulip/zulip_tools.py make_deploy_path)
mv /root/zulip "$deploy_path"
ln -nsf /home/zulip/deployments/next /root/zulip
ln -nsf "$deploy_path" /home/zulip/deployments/next
ln -nsf "$deploy_path" /home/zulip/deployments/current
ln -nsf /etc/zulip/settings.py "$deploy_path"/zproject/local_settings.py
+if [ "$DEPLOYMENT_TYPE" == "dockervoyager" ]; then
+ # We generate the static files later in the entrypoint.sh we don't have the
+ # static files right now, so just create the folder to fix the error with "cp ..."
+ mkdir -p "$deploy_path"/prod-static/serve
+fi
@timabbott
timabbott Mar 19, 2016 Member

Is this still needed with the new model of using the static assets from a release tarball?

@galexrt
galexrt Mar 21, 2016 Contributor

You are right, this sliped through the change back to using the release tarball. I changed now. I'm going to push later.

@timabbott timabbott and 1 other commented on an outdated diff Mar 19, 2016
scripts/setup/postgres-init-db
# Shut down all services to ensure a quiescent state.
-supervisorctl stop all
+# Stop supervisord only if the socket exists, prevents problems with docker
+if [ -e "/var/run/supervisor.sock" ]; then
+ supervisorctl stop all
+fi
@timabbott
timabbott Mar 19, 2016 Member

Can you explain the problem with docker this causes? Is it just supervisorctl throwing an error if it wasn't already running?

@galexrt
galexrt Mar 21, 2016 Contributor

While building the image, supervisor won't be started. That causes supervisorctl stop all to return != 0 and exit the script with an error, this prevent this from happening.

I added a little more explanatory comment to it.

This applies to your other comment on the postgres-init-db file too.

@timabbott timabbott and 1 other commented on an outdated diff Mar 19, 2016
scripts/setup/postgres-init-db
# Shut down all services to ensure a quiescent state.
-supervisorctl stop all
+# Stop supervisord only if the socket exists, prevents problems with docker
+if [ -e "/var/run/supervisor.sock" ]; then
+ supervisorctl stop all
+fi
+
+# Don't "terminate" psql sessions when run in docker!
+if [ "$DEPLOYMENT_TYPE" != "dockervoyager" ]; then
+ # Drop any open connections to any old database. Hackishly call using
+ # source because postgres user can't read /root/zulip/scripts/setup.
+ source "$(dirname "$0")/terminate-psql-sessions" postgres zulip zulip_base
+fi
@timabbott
timabbott Mar 19, 2016 Member

What happens in the docker case in docker -- does this just do nothing there, or what?

@galexrt
galexrt Mar 21, 2016 Contributor

While building the image, supervisor won't be started. That causes supervisorctl stop all to return != 0 and exit the script with an error = build failure, this prevent this from happening.

See my other comment on this file.

@timabbott
timabbott Mar 21, 2016 Member

OK on the supervisor front I think it's reasonable to check for the socket -- that's a correct check for whether supervisord is running.

For the terminate_psql_sessions part, I'm not sure why that would be a problem in the docker setting -- the next lines in this script only work if the script can't access to the database anyway...

@galexrt
galexrt Mar 21, 2016 Contributor

I think that the terminate_psql_sessions will fail, because it tries to connect to the localhost (in case of Docker, psql is not running on the localhost). If the script fails, the "setup" of the image fails and won't start.

I'm going to check if I can remove the if [ "$DEPLOYMENT_TYPE" != "dockervoyager" ]; then here.
I think adding something like in the postgres-init-db, would fix this:

[...]
if [ ! -z "$DB_HOST" ]; then
    POSTGRES_COMMAND="$POSTGRES_COMMAND -h $DB_HOST"
fi
if [ ! -z "$DB_HOST_PORT" ]; then
    POSTGRES_COMMAND="$POSTGRES_COMMAND -p $DB_HOST_PORT"
fi
if [ ! -z "$DB_USER" ]; then
    POSTGRES_COMMAND="$POSTGRES_COMMAND -U $DB_USER"
fi
if [ ! -z "$DB_PASS" ]; then
    export PGPASSWORD="$DB_PASS"
fi
[...]
@timabbott
Member

Sorry for the slow review here, folks! This got caught during the window just before a 2-week vacation, and when I got back I've been working through all the Google Summer of Code traffic on the project. I posted several more comments on this; I think @Galexrt also needs to investigate the emoji not working properly.

@galexrt
Contributor
galexrt commented Mar 21, 2016

@timabbott I'm looking into the emoji problem.

I would like to add the entrypoint.sh from my galexrt/docker-zulip project.
It make things a lot easier, to edit the configs you then could:

  • edit the configs manually
  • use environment variables to automatically set the configuration (=making it "100%" portable)

The entrypoint.sh from my project has some more features, like cron controlled backup, simple backup and restore and automatic certificate generation if no certs are found, it would be easier to maintain one entrypoint.sh and not two.

Would it be okay, if I add it? (even if it is ~629 lines, the features should make up for that many lines 😉)

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Mar 21, 2016
@galexrt galexrt Changes according to the comments from #450 121b837
@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Mar 21, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder

Should fix the Dockerfile-prod problems

Fix documentation around iOS not supporting a custom server.

Fix performance issues with user presence list in large realms.

Whenever a user became active, this triggers an immediate presence
update event (to show that user as active).  The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user.  This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.

[Commit message expanded with more details by Tim Abbott]

Upgrade image to latest package versions before starting

Changes according to the comments from #450
9d84545
@timabbott
Member

We should discuss entrypoint.sh but I'd like to focus effort for now on getting the basic functionality merged first, so we can minimize merge conflict risk, etc.

@galexrt
Contributor
galexrt commented Mar 21, 2016

@timabbott Okay, should I open a fresh pull request with the changes to the script only and add the Docker stuff in this PR?

@timabbott
Member

Nah just keep updating this PR and I'll start merging pieces of it as they're ready...

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Mar 26, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder

Should fix the Dockerfile-prod problems

Fix documentation around iOS not supporting a custom server.

Fix performance issues with user presence list in large realms.

Whenever a user became active, this triggers an immediate presence
update event (to show that user as active).  The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user.  This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.

[Commit message expanded with more details by Tim Abbott]

Upgrade image to latest package versions before starting

Changes according to the comments from #450

Added better but bigger docker entrypoint script

Check if we can remove this

Fixes the emoji problem with the Docker image

Added "smoother" docker-compose.yml
9bc58c6
@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Mar 26, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder

Should fix the Dockerfile-prod problems

Fix documentation around iOS not supporting a custom server.

Fix performance issues with user presence list in large realms.

Whenever a user became active, this triggers an immediate presence
update event (to show that user as active).  The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user.  This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.

[Commit message expanded with more details by Tim Abbott]

Upgrade image to latest package versions before starting

Changes according to the comments from #450

Added better but bigger docker entrypoint script

Check if we can remove this

Fixes the emoji problem with the Docker image

Added "smoother" docker-compose.yml

export the vars for the install script
235c9ec
@galexrt
Contributor
galexrt commented Mar 26, 2016

@timabbott I have now fixed the emoji problem, pushed my entrypoint.sh in and rebased on the latest changes. Is there anything I can still improve?

2016-03-26-124221_1517x1034_scrot
The picture is from my galexrt/docker-zulip, that is now fully included in this PR (it is working because I manually added the files from this pull request/commit to the extracted tarball).

The image currently "can't" be built, because my changes in the scripts, aren't yet in a release tarball. When my changes to the scripts and puppet are in a new release tarball, it'll build without problems (I have tested it with my modified galexrt/docker-zulip image).

This image is now 100% compatible to my galexrt/docker-zulip image. So all users of my image can switch (but no necessary to switch) to the official Zulip image with 0 effort.

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Mar 26, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder

Should fix the Dockerfile-prod problems

Fix documentation around iOS not supporting a custom server.

Fix performance issues with user presence list in large realms.

Whenever a user became active, this triggers an immediate presence
update event (to show that user as active).  The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user.  This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.

[Commit message expanded with more details by Tim Abbott]

Upgrade image to latest package versions before starting

Changes according to the comments from #450

Added better but bigger docker entrypoint script

Check if we can remove this

Fixes the emoji problem with the Docker image

Added "smoother" docker-compose.yml

export the vars for the install script

Merged with the latest `galexrt/docker-zulip` `entrypoint.sh`
5968fe3
@timabbott
Member

Awesome! OOC what was the emoji bug?

I expect to have time to review this in detail in the next week; I plan to start by fully reviewing and merging the install and postgres script changes since those are the most likely to merge conflict with other things folks are doing.

@galexrt
Contributor
galexrt commented Mar 27, 2016

@timabbott The problem was I created the wrong symlinks after the install script. After I removed my code creating links to the prod-static folder it worked perfectly.

BTW the build fail, doesn't seem to be related to my changes.

@timabbott
Member

Yeah the build failure is an issue with how we were getting phantomjs packages; if you rebase onto master the issue is fixed.

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Apr 2, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder

Should fix the Dockerfile-prod problems

Fix documentation around iOS not supporting a custom server.

Fix performance issues with user presence list in large realms.

Whenever a user became active, this triggers an immediate presence
update event (to show that user as active).  The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user.  This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.

[Commit message expanded with more details by Tim Abbott]

Upgrade image to latest package versions before starting

Changes according to the comments from #450

Added better but bigger docker entrypoint script

Check if we can remove this

Fixes the emoji problem with the Docker image

Added "smoother" docker-compose.yml

export the vars for the install script

Merged with the latest `galexrt/docker-zulip` `entrypoint.sh`
ed103d1
@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Apr 2, 2016
@galexrt galexrt This adds a working Dockerfile for "production" use
It includes a `docker-compose.yml` which currently uses my image.
The `docker-entrypoint.sh` is about 285 lines of code, but it has inbuilt database backup and restore function.
(I can remove the backup and restore or add more functions see https://github.com/galexrt/docker-zulip)

I also added an README.docker.md, it should give users a quick start on how to use the docker image.

Added Dockerfile for "production" use
Added other docker related files like docker-compose.yml and docker-entrypoint.sh

Added the generate secrets command to the entrypoint

puppet: Move several debugging tool dependencies out of base.pp.

Rename zulip::postgres_appdb to zulip::postgres_appdb_base.

The purpose of this rename is to allow us to move the postgres-related
configuration out of voyager.pp.

puppet: Move tuned postgres configuration out of voyager.pp.

This should make it easier for someone to run just the tuned Zulip
database on one server and the Zulip frontend on another.

puppet: Move prod-static creation from voyager.pp to app_frontend.pp.

Every app frontend will need this directory and this should help
enable more modular puppet rules.

puppet: Move Zulip apt repository to its own manifest.

puppet: Rename app_frontend.pp to app_frontend_base.pp.

This will enable us to move the remaining app-frontend related content
out of voyager.pp.

puppet: Move memcached into its own puppet module.

puppet: Move memcached and rabbitmq include out of app_frontend_base.pp.

puppet: Move default nginx configuration out of voyager.pp.

Document the puppet configuration somewhat in zulip::voyager.

puppet: Make apt repository conditional on the Ubuntu version.

We still will need to address this in the install script as well.

puppet: Use a variable to configure the postgres version.

puppet: Use $postgres_version in postgres template.

This eliminates hardcoding of the postgres version from the Zulip
puppet configuration.

Fix missing puppet dependencies on postgres package.

Added "custom" dockervoyer puppet manifest

Add the files from the repo directly (dont use git clone to get something we already have in our docker build directory)

Added cron.conf for running cron inside the docker container
Useful for later docker related features like auto backup (see https://github.com/Galexrt/docker-zulip/blob/master/entrypoint.sh#L337-L345)

Filled .dockerignore with directories and fiels to ignore/exclude from the image builds

Fixed the linking of directories

Removed git package and don't add "deb-src" list

Fixed the settings path aka the zproject settings folder

Removed the "removement" of "Service" tasks thing puppet

Remove the puppet files after the puppet install

Removed the zproject directory from the $DATA_DIR
The zproject directory causes trouble if it is just linked. Because zulip uses paths like this for example "deployments/../memcached_prefix".

Removed a trailing slash

Wait for database to come online and create database and schema "manually"
(Shouldn't occure because zulip needs the postgres tsearch thingy extensions, thus rendering every other postgresql image/server than the galexrt/zulip-postgresql-tsearchextras:latest)

Added the wait for database code

Fixed the rabbitmq connection problem

Use sed to replace the default memcached location with the env var MEMCACHED_LOCATION

Added the tsearch_extras extension query to the database (re)creation part

Migrate the database everytime we start (for safety)

Tweaks and fixes to the docker-entrypoint.sh

Restructured the entrypoint.sh

Restructured the docker-entrypoint.sh
This restructure adds database backup and restore functionality
Added "Running in Docker" to the README.md

Fixed the help text for the app:help function

Fixed the restore help line (was app:backup is now app:restore)
Added more explanation to the README.docker.md

Small adjustments and tweaks to the entrypoint.sh

We don't need to copy the uploads from the home/zulip

Added restart always to the docker-compose.yml (because there is no wait for container x to become ready)
Added Try counter at the migration

Tweaks and optimization for the entrypoint.sh

Renamed the zulipgeneratesecrets function

This could fix the exit code 0 problem

Fix for the retry at the database migration

Some changes to the sql queries

Small change to the queries
Fixed the if

Added the forgotten nginx.conf to supervisor
(It's a little bit embarassing, that forgot the main part of this container)

Part 1 of X changes from the comments

Removed the create-admin-user.sh script (see zulip#450 (comment))

Added docker "compatibility" to the postgres-init-db
Use /root/zulip/scripts/setup/postgres-init-db in the docker-entrypoint.sh now

Fixed the problem with the local_settings in the /scripts/lib/install

Don't drop the database when run from docker

Add excpetion for the static files directory in the install script

Fixed the quote in postgres-init-db

Install nodejs using the "Binary Distributions"

Install curl before using it..

apt-get update before installing

Fixed the psql command

Created a third Dockerfile that uses the latest Zulip tarball

Specify the POSTGRES_USER to su into

Weird enough, this fixes the problem with the postgres-init-db script

Use latest Zulip tarball for static-files

Fixed the prod-static order

Forgot to remove the links to the prod-static folder

Should fix the Dockerfile-prod problems

Fix documentation around iOS not supporting a custom server.

Fix performance issues with user presence list in large realms.

Whenever a user became active, this triggers an immediate presence
update event (to show that user as active).  The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user.  This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.

[Commit message expanded with more details by Tim Abbott]

Upgrade image to latest package versions before starting

Changes according to the comments from #450

Added better but bigger docker entrypoint script

Check if we can remove this

Fixes the emoji problem with the Docker image

Added "smoother" docker-compose.yml

export the vars for the install script

Merged with the latest `galexrt/docker-zulip` `entrypoint.sh`
09860e2
@galexrt
Contributor
galexrt commented Apr 2, 2016

I screwed up the rebaseing badly, but I fixed it now, by recreating the branch and adding my modified files. I'm just waiting for Travis to succeed.

@galexrt
Contributor
galexrt commented Apr 9, 2016

Rebased again.

@galexrt
Contributor
galexrt commented Apr 14, 2016

Rebased on latest Zulip changes.

@timabbott Ping.

@timabbott timabbott and 1 other commented on an outdated diff Apr 14, 2016
scripts/lib/install
@@ -1,61 +1,81 @@
#!/usr/bin/env bash
set -xe
+# Specify options for apt
+APT_OPTIONS="${APT_OPTIONS:-}"
+# Install additional packages using apt
+ADDITIONAL_PACKAGES=${ADDITIONAL_PACKAGES:-}
+# Call the default type "voyager", for docker it's "dockervoyager"
+DEPLOYMENT_TYPE="${DEPLOYMENT_TYPE:-voyager}"
+# Can be for example "dockervoyager" or voyager
+VOYAGER_CLASS="${VOYAGER_CLASS:-voyager}"
@timabbott
timabbott Apr 14, 2016 Member

I think we should include the zulip:: in here, and rename it to something like "PUPPET_CLASSES", because this is actually a comma-separated list, not a single value. E.g. you might want "zulip::app_frontend, zulip::memcached"

@galexrt
galexrt Apr 15, 2016 Contributor

@timabbott I renamed the variable, but I think it is better to just "call" the main classes (zulip::voyager or zulip::dockervoyager) here.

@timabbott
Member

Yeah, sorry for the delay reviewing this! If you've been following the project, you've probably seen there's been a ton of new pull requests recently, and I've been prioritizing the quick ones :/

I won't get to do more on this this week, but next week should be a lot better.

@timabbott timabbott referenced this pull request Apr 26, 2016
Closed

Add to DockerHub #698

@timabbott
Member

OK I've finally found some time to look at this some more. I'm getting ready to merge the changes to install and postgres-init-db (have been splitting them into individual readable commits here: https://github.com/timabbott/zulip/commits/docker).

One thing I noticed that I find very confusing is this comment in postgres-init-db:
+# In Docker we have to run this script everytime we start the container, so don't drop.

Why do you need to run postgres-init-db every time you start the container? It seems like ideally this would only ever need to be run once.

@timabbott timabbott commented on an outdated diff Apr 26, 2016
scripts/setup/terminate-psql-sessions
@@ -1,6 +1,5 @@
#!/usr/bin/env bash
-set -e
-set -x
+set -ex
@timabbott
timabbott Apr 26, 2016 Member

We don't need this hunk, right? Can you drop it and the similar ones?

@timabbott timabbott commented on an outdated diff Apr 26, 2016
zerver/lib/queue.py
@@ -260,4 +260,3 @@ def queue_json_publish(queue_name, event, processor):
get_queue_client().json_publish(queue_name, event)
else:
processor(event)
-
@timabbott
timabbott Apr 26, 2016 Member

Can you drop this whitespace change?

@timabbott timabbott commented on the diff Apr 26, 2016
Dockerfile
-USER zulip
-WORKDIR /srv/zulip
+VOLUME ["$DATA_DIR"]
+EXPOSE 80 443
+
+ENTRYPOINT ["/sbin/entrypoint.sh"]
+CMD ["app:run"]
@timabbott
timabbott Apr 26, 2016 Member

Renaming so the production docker file has the "Dockerfile" name is potentially fine, but it might be better to have "Dockerfile-prod" and "Dockerfile.dev" for clarity. Regardless, I think the README.dev.md instructions for using the old development Dockerfile need to be updated to specify the path to the Dockerfile....

@timabbott
Member

I merged the install and postgres-init-db changes that don't check DEPLOYMENT_TYPE, since I think those are all good. Thanks @galexrt!

Still waiting to understand what's going on with how postgres-init-db works in this docker configuration before I try to merge the rest of the changes to those scripts.

@galexrt
Contributor
galexrt commented Apr 27, 2016

@timabbott Sorry for the late response.
In the container the postgres-init-db is currently only run in the install script, that's why there is the exclusion to not run the postgres-init-db script when DEPLOYMENT_TYPE is dockervoyager.

The reason I added those changes to the postgres-init-db script is, so that I can remove the psql queries from the docker-entrypoint.sh and let the postgres-init-db script do the database setup job. I have just now seen that I forgot to replace the queries in the docker-entrpoint.sh. I'm going to push this change with the next rebased commit.

I'm going to add the improvements and fixes you pointed out in the comments, in that commit too.

@galexrt
Contributor
galexrt commented Apr 27, 2016

@timabbott I have now pushed the changes from the comments.
I'm currently rebasing on the master branch.

@galexrt
Contributor
galexrt commented Apr 27, 2016

@timabbott I just pushed my rebased docker branch. I had readded my changes to the postgres-init-db, but now after checking in the docker-entrypoint.sh I have now reverted the postgres-init-db back to the state in your docker branch.

@timabbott timabbott and 1 other commented on an outdated diff Apr 27, 2016
README.dev.md
@@ -380,7 +380,7 @@ Then you should create the Docker image based on Ubuntu Linux, first
go to the directory with the Zulip source code:
```
-docker build -t user/zulipdev .
+docker build -f Dockerfile-dev -t user/zulipdev .
@timabbott
timabbott Apr 27, 2016 edited Member

Does the user need to pass -f Dockerfile-dev when using docker run and other commands as well?

@galexrt
galexrt Apr 27, 2016 edited Contributor

The -f Dockerfile-dev is only needed for the docker build command and not for the docker run commands.

@timabbott timabbott added this to the 2016 roadmap milestone Apr 29, 2016
@timabbott
Member

I'm really really sorry @galexrt, but I haven't had a chance to work on this more before a vacation for 2 weeks I'm leaving for tomorrow. This is the pull request that I've been most behind on finding time to review, so I'll try hard to find time to look in detail when I get back.

@galexrt
Contributor
galexrt commented May 19, 2016

No problem. I'm going to keep the PR rebased as best as I can. Have a good vacation!

@galexrt
Contributor
galexrt commented Jul 12, 2016

I don't see why the backend build is failing because this comment doesn't change any backend stuff..

@timabbott
Member

If you click on "Details" you can actually read the error messages. The backend suite also runs the Zulip linters; the failure is:

+./tools/lint-all
No newline at the end of file.  Fix with `sed -i '$a\' docker-entrypoint.sh`
The command "./tools/travis/$TEST_SUITE" exited with 1.
@galexrt
Contributor
galexrt commented Jul 12, 2016

@timabbott Thanks for the heads up!
I just added a new line to the docker-entrypoint.sh.

@timabbott
Member

@galexrt I did a bit of related work that I wanted to point you to in 0219149 (currently on my branch but should be merged soon); see #1235 for how to use it. This should help with eventually making it possible to have a set of docker files to run each Zulip service in its own instance.

@galexrt
Contributor
galexrt commented Jul 12, 2016

@timabbott That's awesome!
Could you please mention me when the changes are merged so that I can modify the Dockerfile to match your changes.
The dockervoyager.pp is still needed with your changes in mind. Is that okay?

@timabbott
Member

Yep, can do and totally agreed that we'll still need dockervoyager.pp.

@timabbott timabbott commented on the diff Jul 25, 2016
.dockerignore
@@ -0,0 +1,11 @@
+.git/
+.gitattributes
+.gitignore
+.travis.yml
+changelog.md
+LICENSE
+README*
+THIRDPARTY
+Vagrantfile
+frontend_tests/
+docker-compose*.yml
@timabbott
timabbott Jul 25, 2016 Member

Would it work to just make this a symlink to .gitignore?

@timabbott
timabbott Jul 25, 2016 edited Member

If not, can you add a comment at the top of this explaining how one should decide which things need to go in here? that will help a lot with our being able to maintain this effective. E.g. if the intent is to exclude files that aren't need in production, .gitattributes (used in maintaining our release tarballs) is the authoritative source of that info, and it'd be great to have the Docker system use that (though I'm not sure the tools support it, so we might have to just do it via copy).

@galexrt
galexrt Feb 13, 2017 Contributor

Comment added.

@timabbott timabbott commented on an outdated diff Jul 25, 2016
README.docker.md
@@ -0,0 +1,56 @@
+Zulip
@timabbott
timabbott Jul 25, 2016 Member

Can you move this to somewhere under docs/ as part of our ReadTheDocs tree?

@timabbott timabbott commented on an outdated diff Jul 25, 2016
README.dev.md
+To stop the server use:
+```
+docker ps
+docker kill <container id>
+```
+
+If you want to run all the tests you need to start the servers first,
+you can do it with:
+
+```
+docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash
+$ tools/test-all-docker
+```
+
+You can modify the source code in your development machine and review
+the results in your browser.
@timabbott
timabbott Jul 25, 2016 Member

I think these changes are a mis-resolved merge conflict with our moving this content to docs/install-docker-dev.md. Can you address that?

@timabbott
Member

@galexrt I'd like to merge some more pieces of this. Here's a proposed way to split this up into mergable commits:

  • A commit that just renames the existing Dockerfile to Dockerfile.dev and updates the existing development docs accordingly
  • A commit that just adds the puppet stuff (dockervoyager.pp and its dependencies).
  • A commit adding the .dockerignore file
  • A commit adding Dockerfile, plus the documentation on how to use this.
@timabbott timabbott and 1 other commented on an outdated diff Jul 25, 2016
Dockerfile-prod
@@ -0,0 +1,27 @@
+FROM quay.io/sameersbn/ubuntu:latest
+MAINTAINER Alexander Trost <galexrt@googlemail.com>
+
+ENV ZULIP_VERSION="1.3.10" DATA_DIR="/data"
@timabbott
timabbott Jul 25, 2016 Member

For ZULIP_VERSION, would it make sense to use zulip-server-latest.tar.gz by default but have a way for the user to pass in a different version as an override? Then we don't need to update this when we do Zulip releases.

@galexrt
galexrt Jul 26, 2016 Contributor

I would just change the ZULIP_VERSION to latest so it always downloads and installs the latest version or the user specified.

@timabbott timabbott commented on an outdated diff Jul 25, 2016
Dockerfile-prod
@@ -0,0 +1,27 @@
+FROM quay.io/sameersbn/ubuntu:latest
+MAINTAINER Alexander Trost <galexrt@googlemail.com>
+
+ENV ZULIP_VERSION="1.3.10" DATA_DIR="/data"
+
+RUN apt-get -q update && \
+ apt-get -q dist-upgrade -y && \
+ mkdir -p "$DATA_DIR" /root/zulip && \
+ wget -q "https://www.zulip.com/dist/releases/zulip-server-$ZULIP_VERSION.tar.gz" -O /tmp/zulip-server.tar.gz && \
+ tar xfz /tmp/zulip-server.tar.gz -C /root/zulip --strip-components=1 && \
+ rm -rf /tmp/zulip-server.tar.gz && \
+ export VOYAGER_CLASS="dockervoyager" DEPLOYMENT_TYPE="dockervoyager" \
+ ADDITIONAL_PACKAGES="python-dev python-six python-pbs" has_nginx="0" has_appserver="0" && \
@timabbott
timabbott Jul 25, 2016 Member

python-pbs is no longer used in Zulip, so you can probably drop it here.

@timabbott timabbott and 1 other commented on an outdated diff Jul 25, 2016
Dockerfile-prod
@@ -0,0 +1,27 @@
+FROM quay.io/sameersbn/ubuntu:latest
+MAINTAINER Alexander Trost <galexrt@googlemail.com>
+
+ENV ZULIP_VERSION="1.3.10" DATA_DIR="/data"
+
+RUN apt-get -q update && \
+ apt-get -q dist-upgrade -y && \
+ mkdir -p "$DATA_DIR" /root/zulip && \
+ wget -q "https://www.zulip.com/dist/releases/zulip-server-$ZULIP_VERSION.tar.gz" -O /tmp/zulip-server.tar.gz && \
+ tar xfz /tmp/zulip-server.tar.gz -C /root/zulip --strip-components=1 && \
+ rm -rf /tmp/zulip-server.tar.gz && \
+ export VOYAGER_CLASS="dockervoyager" DEPLOYMENT_TYPE="dockervoyager" \
+ ADDITIONAL_PACKAGES="python-dev python-six python-pbs" has_nginx="0" has_appserver="0" && \
+ /root/zulip/scripts/setup/install && \
+ chown zulip:zulip /opt/createZulipAdmin.sh && \
@timabbott
timabbott Jul 25, 2016 edited Member

where does createzulipadmin.sh come from? Maybe you need to add that file to this commit series?

@galexrt
galexrt Jul 26, 2016 Contributor

It is a script that creates a zulip user with admin perms. The script is from my galexrt/docker-zulip repo.
Should I include it so Docker users have a simple way of creating an user?

@timabbott timabbott and 1 other commented on an outdated diff Jul 26, 2016
puppet/zulip/manifests/dockervoyager.pp
+# Through this split of services, it is easier to scale the services to the needs.
+class zulip::dockervoyager {
+ include zulip::base
+ # zulip::apt_repository must come after zulip::base
+ include zulip::apt_repository
+ include zulip::app_frontend
+ $appdb_packages = [# Needed to run process_fts_updates
+ "python-psycopg2",
+ ]
+ define safepackage ( $ensure = present ) {
+ if !defined(Package[$title]) {
+ package { $title: ensure => $ensure }
+ }
+ }
+ safepackage { $appdb_packages: ensure => "installed" }
+ include zulip::supervisor
@timabbott
timabbott Jul 26, 2016 Member

is there a reason zulip::supervisor is down here?

@galexrt
galexrt Jul 26, 2016 Contributor

Not really. I'm moving it up.

@timabbott
Member

Posted a few comments on various things. The big issue here is that I'm still not sold on how much duplicated code docker-entrypoint.sh has -- e.g. why can't bootstrapRabbitMQ just shell out to scripts/setup/configure-rabbitmq (perhaps with some new options added to support making the usernames/etc. variables, and similarly for the initial database setup)? At present, it seems like there'd just going to be maintenance work involved with updating that script every time we refactor Zulip's setup process, which could be avoided. But perhaps I'm missing something here.

@galexrt
Contributor
galexrt commented Jul 26, 2016

@timabbott I'm going to look into de-duplicate the code from the entrypoint by adding some changes to the configuration scripts of Zulip in a separate PR.

@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Jul 29, 2016
@galexrt galexrt Deduplication of configuration: Rabbitmq
This is for the deduplication of setup code, see PR #450
d79d5bc
@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Jul 29, 2016
@galexrt galexrt Deduplication of configuration: Rabbitmq
This is for the deduplication of setup code, see PR #450

Let's try to use the hostname as the rabbitmq host
1d3e947
@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Aug 4, 2016
@galexrt galexrt Deduplication of configuration: Rabbitmq
This is for the deduplication of setup code, see PR #450

Let's try to use the hostname as the rabbitmq host

Check if supervisor socket exists before restarting

One variable for the host + username in rabbitmq setup
c4d4868
@galexrt galexrt added a commit to galexrt/zulip that referenced this pull request Aug 4, 2016
@galexrt galexrt Deduplication of configuration: Rabbitmq
This is for the deduplication of setup code, see PR #450

Let's try to use the hostname as the rabbitmq host

Check if supervisor socket exists before restarting

One variable for the host + username in rabbitmq setup
0a7386e
@galexrt
Contributor
galexrt commented Sep 17, 2016 edited

I finally had some time to rebase and remove the rabbitmq and initialize-database duplicated code from the entrypoint.

For the rabbitmq stuff to work I need the PR #1464.

@timabbott timabbott modified the milestone: Zulip roadmap, Old roadmap Nov 18, 2016
@galexrt
Contributor
galexrt commented Jan 18, 2017 edited

I rebased the commits.
There is one thing that still causes problems.
The supervisor service fails to start on certain Docker build environments (like Docker Hub), because the supervisor service can't start. Is there a way to not start the supervisor service in puppet when running in a container?

@galexrt
Contributor
galexrt commented Jan 22, 2017

Fixed the docs.


@timabbott Any idea how I can stop Puppet from restarting the Supervisor service when in running in Docker build?

@timabbott
Member

Sorry for the slow reply; can you explain the context/problem a bit more @galexrt? Is this puppet running inside a container that is running the main app (and thus has supervisord included) or one of the other containers?

@galexrt
Contributor
galexrt commented Feb 8, 2017

The problem is that when Puppet tries to start supervisor service it fails. Resulting in the whole deployment to fail with a non zero exit code.
I may have found a way to fix this.

@galexrt
Contributor
galexrt commented Feb 8, 2017 edited

I have added a variable named $ignoreSupervisorService to "create" the supervisor service but with /bin/true as the start/stop handlers.
This fixes the issues with supervisor in the Puppet deployment not starting when building the Docker image.
Is this okay?


One other thing that is on my mind, to make building the Docker image simple for users. Is that the Dockerfile-prod should be renamed to Dockerfile and the current Dockerfile should be renamed to Dockerfile-dev.

@galexrt
Contributor
galexrt commented Feb 12, 2017

@timabbott Puppet lint doesn't show me any errors, but Travis lint-all fails with:

Error: Could not parse for environment production: Syntax error at ';'; expected '}' at /home/travis/build/zulip/zulip/puppet/zulip/manifests/dockervoyager.pp:21

Is that a syntax error or not?

@galexrt
Contributor
galexrt commented Feb 20, 2017 edited

@timabbott Could you tell me about the error I got in the Travis build? I really want to finally get this into Zulip finally.

Edit: I now renamed the Dockerfiles "again", as some experience shows, that users will just clone the repo and run docker build on it and expect a full working image to come out of it.

@galexrt galexrt This adds a working Dockerfile
Added newline to docker-entrypoint
Because of travis lint-all failing

Added the new install env vars
The changes in #1254 are required for this to work.

Added the improvements from the comments

Renamed the docker-compose
Moved my docker README to the docs folder and added some changes to it

Removed Rabbitmq duplicated code
Added IGNORE_RABBITMQ_ERRORS to ignore may occuring errors

Updated the docker-entrypoint with the latest bug fixes

Removed duplicate initialize-database code

Moved the supervisor service down a bit

Corrected the docker entrypoint perms

Updated dockerignore

Small formatting changes to fix puppet-lint errors

FIxed the docs

Added variable to toggle if supervisor service should be "ignored"

Replaced domain with example.com

Added comment to the .dockerignore

Made the .dockerignore a bit more clear

Refined the dockerignore

Synced docker-entrypoint with galexrt/docker-zulip entrypoint

Renamed the Dockerfiles to allow "new" users to simply build the correct
image
b374e4e
@timabbott
Member

Travis CI just changed their build environments overnight, it's not your issue. Working on it.

@jezdez
jezdez commented Feb 21, 2017

@timabbott @galexrt Hey there, I've been following along the effort to make Zulip play better with Docker and wondered if you considered using one of the official Postgres images instead of the fork of a custom one that @galexrt has been using for a while now.

I'm asking since I'm interested in the long term maintenance of a Docker based deployment of Zulip where a wholly custom Postgres Docker image could be a risk and lead to the increase of TCO. I realize that this PR has been coming for a long time, apologies for this late feedback, @galexrt. 🙇

Just out of curiosity I checked the diff between @galexrt's Postgres fork and the main repo of the custom Postgres repo and found it not too complicated to adopt to the official Postgres docker image. Luckily the image is able to be extended fairly easy (see "How to extend this image").

So I think it boils down to:

  • installing the tsearch_extras package
  • hooking up the dictionary and stop words
  • add a shell script that adds the zulip schema and enables unaccent and tsearch_extras extions

The result is https://github.com/jezdez/postgres-zulip-tsearchextras if you want to take a look at. If this is hooked up with CircleCI or Travis CI to push to Docker Hub whenever @timabbott does a release of tsearch_extras I think this should be the most stable solution going forward. What do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment