Deployment

[stas]

Hi,
we have an issue with the deployment of the app.
Basically, the way static files are handled right now, it's a pain!

Even though the app, bundles the static files and builds them, there's no easy way to get those files out of the container in a sane way.

• We either need a proper CDN/static host
• Or we need to automate the static file build process

Also currently the overall deployment process is not smooth at all (and me having access to the server with all the websites doesn't make me sleep better, if you know what I mean 🛐 )

I dream of having an automated build process, one command step. Something like Heroku would be lovely!

This is an open conversation btw...
Thanks

/cc @rysiekpl @pudo

[stas]

/cc @sungraze

[rysiekpl]

So.

Okay, let's fix the static files handling then. What do you mean by:

there's no easy way to get those files out of the container in a sane way

Why would we need to get the files out of the docker image at all? The idea of using images is so that they're self-contained, after all.

As far as this is concerned:

and me having access to the server with all the websites doesn't make me sleep better, if you know what I mean

...you do not have access to the production server. You have access to the testing server. Yes, there are other sites there, but backups are made daily and if anything goes wrong we can just revert stuff from the backup.

Besides, @czesiekhaker is looking at ways of automating build on testing and production servers using a CI pipeline, but since it was not a priority up until now, it takes a bit of time. However, we need it for Weaving too, and potentially for other things, so it's actually a good thing we needed to dive into this now.

The idea is for deploys to happen automagically after pushing to a given branch (say, master or staging, or whatever we want to call it).

[stas]

Great,
thanks for reviewing my feedback so far.

As mentioned before, I don't actually want to push for CI/CD. Tbh, I don't think OCCRP projects are there (most of the apps do not have tests or any sort of automation from what I know). I'd rather just have an automated provisioning process with a simple standard deployment method that doesn't include access to OS. A great example for my case is dokku.

[rysiekpl]

The CI pipeline will initially be used for deployment that indeed does not include access to OS. Once we have that up and running, we can later start using it for tests, once they start being written.

I understand Heroku is familiar and convenient to you, but we need to review if it makes sense to set-up Dokku just to deploy ID2, or perhaps if it will be useful for our other software.

That being said, while I sympathize with you on how annoying it must be to have to log-in to a server and issue 3 commands to deploy your code, I hope you can make this sacrifice for us while we look at how to set-up deploys in the long run.

Thanks.

[stas]

What about the provisioning? If we will have to ask you every time to set the CI/CD for us than we are back to where we are now (like KEYCLOACK keys, database rollbacks/resets, new vhosts etc.)
I find this actually quite time-consuming right now.

[rysiekpl]

It is time-consuming and we need to automate this, yes. Reason why it hasn't yet is that 2 years ago Tech Team was 3 people. It was not worth it to invest time into that amount of automation. We've grown, we need to own up to that.

Some things can be (semi-)easily automated, some not that much. For example, as far as Keycloak is concerned, while we can have test realms where developers have full admin privileges, obviously production will always be a bit of a different world.

Database rollbacks/resets (and perhaps ) certainly can and should be automated, at least in testing/staging. New vhosts -- that's tricky due to TLS. We can automate certificate issuance (again, on the ToDo list), but changes to nginx configs will have to stay manual at least for some time.

I know that this environment is more rigid than optimal for quick development, but it is so not without reasons. There are very concrete security considerations we have to take into account. We are not a random startup on Teh Intertubes building an smart juicer, we're dealing with data that can potentially put sources in jeopardy. And we have to think not only about data "leaking", but also being subpoenaed. That's why we avoid many of the convenient options, like Heroku.

Finally, the devops/sysops team has to grow, and is already growing.

Having said all that, perhaps I am missing something, but I do not see a lot of new services being provisioned. We're not even talking 3 new vhosts per month. So I have to ask, are we expecting a sudden spike in provisioned services?

[stas]

I added some instructions to the readme file.
https://github.com/occrp/investigative-dashboard-2#in-production

I'm leaving this ticket up to @rysiekpl

[rysiekpl]

Ah, great! Thank you! :)

By the way, as far as static resources are concerned, this:

$ docker-compose run web python manage.py collectstatic
$ docker-compose run web python manage.py assets build

...could be moved to an entrypoint script. Or, if it doesn't touch any resources that only become available upon running the container (volume-mounted directories, database, etc), perhaps even into the Dockerfile.

[stas]

Yeah, we really need to have the volumes mounted before these two run.
This is because nginx is looking for static files in the mounted volume path.

[rysiekpl]

Right. Here's a first jab at collectstatic/assets build in entrypoint script: #44

I see that some of these are done already in the Dockerfile, right? Perhaps, since we need to do it at runtime anyway, it would make sense to remove that from the Dockerfile, thus shortening the build time.

[stas]

@rysiekpl I have some bad news. After we switched to the entrypoint feature, the deployment is now affected by docker/compose#3140 :(

[rysiekpl]

Why can't we have the CMD set in the docker-compose.yml too? Also, why do we set entrypoint in docker-compose.yml instead of in Dockerfile?

[stas]

Please see the upstream issue. It looks like it just doesn't work as expected. My biggest issue is when you try running docker-compose run web <CMD>, it doesn't run the CMD. I spent yesterday some time with it, and I decided to raise the question here :(

[rysiekpl]

Why are you using docker-compose run? That's for one-offs. You want to use docker-compose up, or even docker-compose up -d (so that it goes into background).

I understand the upstream issue. What I am saying is: you can have the command set in the docker-compose.yml file, along with the entrypoint. That way even if setting the entrypoint removes the default command (set in the Dockerfile), the command is explicitly set in the docker-compose.yml file anyway, so it still works.

[rysiekpl]

Ok, I don't get it. I see the command is set in both docker-compose.*.yml files:

• https://github.com/occrp/investigative-dashboard-2/blob/master/docker-compose.dev.yml#L22
• https://github.com/occrp/investigative-dashboard-2/blob/master/docker-compose.prod.yml#L17

...and that no entrypoint is explicitly set in these files. How is this related to the issue you linked to? What exactly are you trying to do and what is the problem?

[stas]

Why are you using docker-compose run?

Migrations.

I understand the upstream issue. What I am saying is: you can have the command set in the docker-compose.yml file, along with the entrypoint. That way even if setting the entrypoint removes the default command (set in the Dockerfile), the command is explicitly set in the docker-compose.yml file anyway, so it still works.

The version from master with the command set, but no entrypoint, didn't work for me. It would just not run it. I left the uncommitted changes on woodward. This is how I managed to start the container.

@rysiekpl sorry for all the headache, but I'm just reporting what I find. Trust me, there's nothing magical that I'm trying to do here :|

[rysiekpl]

Right.

# docker-compose run --entrypoint /bin/bash web -c 'python manage.py migrate'
Starting Investigative Dashboard, v. 2.4.0 (production)
Operations to perform:
  Apply all migrations: accounts, activity, admin, api_v3, auth, captcha, contenttypes, core, databases, oauth2_provider, registration, sessions, social_auth, ticket
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

Is this something on-topic? Should we add python manage.py migrate to the entrypoint script so that it gets executed automagically each time docker-compose up is run?

In general if you need to run a command inside the container, use:

docker-compose run --entrypoint /bin/bash web -c 'some-commad arg1 arg2 (...)'

For example:

# docker-compose run --entrypoint /bin/bash web -c 'python manage.py migrate'
Starting Investigative Dashboard, v. 2.4.0 (production)
Operations to perform:
  Apply all migrations: accounts, activity, admin, api_v3, auth, captcha, contenttypes, core, databases, oauth2_provider, registration, sessions, social_auth, ticket
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
root@woodward /srv/tools/id # docker-compose run --entrypoint /bin/bash web -c 'python manage.py makemigrations'
Starting Investigative Dashboard, v. 2.4.0 (production)
Migrations for 'api_v3':
  api_v3/migrations/0006_auto_20170724_1035.py:
    - Alter field upload on attachment
    - Alter field country on ticket
    - Alter field kind on ticket
    - Alter field request_type on ticket
    - Alter field status on ticket
Migrations for 'ticket':
  ticket/migrations/0002_auto_20170724_1035.py:
    - Alter field country on companyticket
Migrations for 'databases':
  databases/migrations/0002_auto_20170724_1035.py:
    - Alter field country on externaldatabase
    - Alter field db_type on externaldatabase
Migrations for 'accounts':
  accounts/migrations/0009_auto_20170724_1035.py:
    - Change managers on profile
    - Alter field country on profile
    - Alter field groups on profile
    - Alter field last_login on profile
Migrations for 'activity':
  /usr/local/lib/python2.7/dist-packages/activity/migrations/0002_auto_20170724_1035.py:
    - Alter field action_object_id on action
    - Alter field actor_object_id on action
    - Alter field is_public on action
    - Alter field target_object_id on action
    - Alter field timestamp on action

[rysiekpl]

For the record, there was a problem with the command not being passed to the entrypoint.sh script, because docker treats ENTRYPOINT /id/entrypoint.sh and ENTRYPOINT ["/id/entrypoint.sh"] differently.

This has been fixed by 5957fde

[stas]

With the adoption of Rancher this is no longer an issue.