Updates sidecar setup and run #199

pcrespov · 2018-08-24T20:03:24Z

What do these changes do?

Upgrades sidecar setup and module structure and fixes some issues.

Missing setup.py
Missing requirement-dev.txt
Non-root user in docker.
venv in docker
rearrange package structure: main.py, tasks.py, celery.py and utils.py
entrypoint script to start as root and run as scu

Extra fixes on webserver

retreived config files as resources
modified Dockerfile: should replicate the way we test in source
- keeps same relative folder structure
- all 3erd, 2nd and main packages install in a venv
- runs as a non-root user
- boot script

Related issue number

closes Create setup for sidecar #198

Checklist

I think the code is well written
~~Unit tests for the changes exist~~
~ Documentation reflects the changes~
If you design a new module, add your user to .github/CODEOWNERS

@mguidon, @sanderegg

- created setup and requirements for dev&prod - installs package in docker (instead of reference via sys.path). this way all third-parties are installed. - minor cleanup of code

- added a boot.sh script to start celery worker - restructured package submodules. Otherwise got error (see comments in ITISFoundation#198) -

- all volumes bound to /home/scu - All modules pip installed (in dev w/ edit mode). - All files produce in dev mode on bound volumes are not deletable - Production stage is further optimized by taking only venv from base - Fixes sidecar access to input/output/log volumes

…env and pip installing 3rd parties

…duction-build stage - Normalized naming of package and console script

- removed usused cs_s4l from config - added env for rabbit

- cleanup Dockerfile - minor doc and logs

services/docker-compose.devel.yml

sanderegg · 2018-08-30T09:44:34Z

services/sidecar/Dockerfile

+VOLUME /home/scu/input
+VOLUME /home/scu/output
+VOLUME /home/scu/log
+#VOLUME /var/run/docker.sock DO NOT EXPOSE .. fails!


probably not needed here?

services/web/server/src/server/settings/config.py

sanderegg

So, after calling
make build
I get an error that DOCKER_GID is not defined

commit 0df033b Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Thu Aug 30 14:21:44 2018 +0200 Review feedback: removed unnecessary config sections commit bab7831 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Thu Aug 30 13:59:07 2018 +0200 Fixes missing dlls commit c0bc7d6 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Thu Aug 30 11:53:41 2018 +0200 Fixes access to docker socket: - cleanup Dockerfile - minor doc and logs commit 5b6f4c7 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 17:47:13 2018 +0200 Minor cleanup and doc commit 43f7305 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 15:52:30 2018 +0200 Minor change in web server commit 7f07022 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 15:52:16 2018 +0200 sidecar: starts as root (via entrypoint) but runs as scu (non-root) commit 90359c4 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 15:51:32 2018 +0200 Fixes sidecar: temporary fix to connection to other services commit 7619671 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 15:44:42 2018 +0200 Fixes host in server running inside a docker commit bdbf078 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 15:43:56 2018 +0200 Separating maintenance services commit 83e492d Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 13:46:55 2018 +0200 Fixes on web server: - removed usused cs_s4l from config - added env for rabbit commit 8fee402 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Wed Aug 29 11:48:46 2018 +0200 Fixes on webserver: - Tmp solution for production target based on production-build stage - Normalized naming of package and console script commit a5d8220 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Tue Aug 28 22:06:59 2018 +0200 Implements entry in ITISFoundation#186: - upgrades Dockerfile with .venv and pip installing 3rd parties commit 3c55bf3 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Tue Aug 28 18:46:57 2018 +0200 Setting up boot for web service and forgot configs commit 8003744 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Tue Aug 28 18:40:12 2018 +0200 Adding config data and retreiving it as resource commit 94de3d7 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Tue Aug 28 10:58:02 2018 +0200 Minor changes in docker after discussion with MaG commit e663a1e Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Mon Aug 27 22:08:38 2018 +0200 Updated ownership commit bf486b3 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Mon Aug 27 21:53:12 2018 +0200 Fixes linting errors and added notes commit 752c7bf Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Mon Aug 27 19:49:39 2018 +0200 WIP ITISFoundation#198: non-root user - all volumes bound to /home/scu - All modules pip installed (in dev w/ edit mode). - All files produce in dev mode on bound volumes are not deletable - Production stage is further optimized by taking only venv from base - Fixes sidecar access to input/output/log volumes commit ec755bc Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Fri Aug 24 21:53:13 2018 +0200 WIP ITISFoundation#198: - added a boot.sh script to start celery worker - restructured package submodules. Otherwise got error (see comments in ITISFoundation#198) - commit 9bbfbde Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Fri Aug 24 21:52:12 2018 +0200 Minor update commit 53c7707 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Fri Aug 24 20:29:26 2018 +0200 Fixes import failure of trafaret and updages vs sqalquemy commit a27a279 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Fri Aug 24 20:24:49 2018 +0200 WIP ITISFoundation#198: setup for sidecar - created setup and requirements for dev&prod - installs package in docker (instead of reference via sys.path). this way all third-parties are installed. - minor cleanup of code commit bbc7cb6 Author: Pedro Crespo <crespo@itis.ethz.ch> Date: Fri Aug 24 20:22:31 2018 +0200 Minor: sync req versions

mguidon

Pair review done

sanderegg · 2018-08-31T06:42:46Z

so this does not work on Windows. There is no docker group so anyone mounting the docker socket as standard user is not able to access the docker engine when running from a windows host.

Pedro Crespo added 17 commits July 6, 2018 14:34

Fixes typos in client after bad merge

2741834

Merge remote-tracking branch 'upstream/master'

9646035

Merge remote-tracking branch 'upstream/master'

c3ad81c

Merge remote-tracking branch 'upstream/master'

d7b91f7

Merge remote-tracking branch 'upstream/master'

a65aa8e

Merge remote-tracking branch 'upstream/master'

4edc12a

Merge remote-tracking branch 'upstream/master'

460bdea

Merge remote-tracking branch 'upstream/master'

408990d

Merge remote-tracking branch 'upstream/master'

ee1b91b

Merge remote-tracking branch 'upstream/master'

cf7b3f4

Merge remote-tracking branch 'upstream/master'

3f4eed8

Merge remote-tracking branch 'upstream/master'

14cbec2

Minor: sync req versions

bbc7cb6

WIP ITISFoundation#198: setup for sidecar

a27a279

- created setup and requirements for dev&prod - installs package in docker (instead of reference via sys.path). this way all third-parties are installed. - minor cleanup of code

Fixes import failure of trafaret and updages vs sqalquemy

53c7707

Minor update

9bbfbde

WIP ITISFoundation#198:

ec755bc

- added a boot.sh script to start celery worker - restructured package submodules. Otherwise got error (see comments in ITISFoundation#198) -

ghost assigned pcrespov Aug 24, 2018

ghost added the review label Aug 24, 2018

pcrespov changed the title ~~WIP - Is#198/sidecar~~ WIP - is#198/sidecar Aug 24, 2018

pcrespov requested review from mguidon and sanderegg August 24, 2018 20:03

pcrespov changed the title ~~WIP - is#198/sidecar~~ is#198/sidecar: Updates sidecar setup and run Aug 27, 2018

Fixes linting errors and added notes

bf486b3

sanderegg mentioned this pull request Aug 27, 2018

(SimCoreUser) will be the new user for host&container in order to avoid being root. ID with a very high number. Mechanism that the user in the host and in docker image are the same one. #90

Closed

4 tasks

Pedro Crespo added 4 commits August 27, 2018 22:08

Updated ownership

e663a1e

Minor changes in docker after discussion with MaG

94de3d7

Adding config data and retreiving it as resource

8003744

Setting up boot for web service and forgot configs

3c55bf3

Implements entry in ITISFoundation#186: - upgrades Dockerfile with .v…

a5d8220

…env and pip installing 3rd parties

pcrespov changed the title ~~is#198/sidecar: Updates sidecar setup and run~~ Updates sidecar setup and run Aug 29, 2018

Pedro Crespo added 7 commits August 29, 2018 11:54

Fixes on webserver: - Tmp solution for production target based on pro…

8fee402

…duction-build stage - Normalized naming of package and console script

Fixes on web server:

83e492d

- removed usused cs_s4l from config - added env for rabbit

Separating maintenance services

bdbf078

Fixes host in server running inside a docker

7619671

Fixes sidecar: temporary fix to connection to other services

90359c4

sidecar: starts as root (via entrypoint) but runs as scu (non-root)

7f07022

Minor change in web server

43f7305

pcrespov mentioned this pull request Aug 29, 2018

sidecar: review setup and start states #206

Closed

Pedro Crespo added 2 commits August 29, 2018 17:50

Minor cleanup and doc

5b6f4c7

Fixes access to docker socket:

c0bc7d6

- cleanup Dockerfile - minor doc and logs

sanderegg reviewed Aug 30, 2018

View reviewed changes

Pedro Crespo added 2 commits August 30, 2018 13:59

Fixes missing dlls

bab7831

Review feedback: removed unnecessary config sections

0df033b

pcrespov added this to the Libra milestone Aug 30, 2018

sanderegg requested changes Aug 30, 2018

View reviewed changes

Fixed volume map between sidecar and workcar containers

4c316eb

mguidon approved these changes Aug 30, 2018

View reviewed changes

sanderegg approved these changes Aug 31, 2018

View reviewed changes

pcrespov merged commit 89c582c into ITISFoundation:master Aug 31, 2018

ghost removed the review label Aug 31, 2018

pcrespov deleted the is#198/sidecar branch September 18, 2018 08:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates sidecar setup and run #199

Updates sidecar setup and run #199

pcrespov commented Aug 24, 2018 •

edited

Loading

sanderegg Aug 30, 2018

sanderegg left a comment

mguidon left a comment

sanderegg commented Aug 31, 2018 •

edited

Loading

Updates sidecar setup and run #199

Updates sidecar setup and run #199

Conversation

pcrespov commented Aug 24, 2018 • edited Loading

What do these changes do?

Related issue number

Checklist

sanderegg Aug 30, 2018

Choose a reason for hiding this comment

sanderegg left a comment

Choose a reason for hiding this comment

mguidon left a comment

Choose a reason for hiding this comment

sanderegg commented Aug 31, 2018 • edited Loading

pcrespov commented Aug 24, 2018 •

edited

Loading

sanderegg commented Aug 31, 2018 •

edited

Loading