New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerify all the things #3296

Closed
msmith-techempower opened this Issue Feb 16, 2018 · 8 comments

Comments

Projects
None yet
4 participants
@msmith-techempower
Member

msmith-techempower commented Feb 16, 2018

Abstract

Move the suite away from the current methodology of using scripting files to handle dependencies, installation into the $IROOT directory for sandboxing, and instead use Docker CE.

Reasoning

  1. Installing software to $IROOT is hard (much harder than apt-get) and prone to difficult-to-identify errors
  2. Many dependencies use apt-get already for various reasons, and migrating away from globally installed software is hard (much harder than allowing apt-get)
  3. Docker provides a much cleaner reusable sandbox environment that has been fire-tested much more-so than our in-house implementation
  4. In testing, Docker does not appear to incur much (if any) performance penalty

Ongoing work is in this branch.

Action Items (in no particular order)

    • Create base tfb dockerfile with as minimal software installed as possible #3292
    • Implement base dockerfile to have an entrypoint of TFBReapear so that everything is run through him #3292
    • Create POC using only gemini and the dockerfile(s) required to build and run in console mode #3292
    • Create POC suite implementation using the dockerfile(s) require to build and run gemini and perform a verification #3292
    • Implement stable mechanism for finding running docker containers and stopping them #3292
    • Implement stable mechanism for removing docker images for a specific test after the container is stopped (there is no reason to keep a gemini image after we have run the gemini test) #3292
    • Create docker folder hierarchy that would allow for similar grouping to our current dependency hierarchy (docker/languages, docker/frameworks, etc) #3316
    • Change benchmark_config.json to specify a dockerfile instead of a setup.sh #3292
    • Implement database mechanisms currently handled by fw_depends (which is being deprecated and removed with the docker work) #3292
    • Implement client mechanisms currently handled by fw_depends (installing wrk on the remote client, etc) #3292
    • Update vagrant install to call tfb --init --quiet to initialize the software on the database and client machine (localhost) #3292
    • Fix docker build process to split the output to the console as well as the log (see framework_test.py for an example of this; this is needed to see output in Travis-CI, for example) #3292
    • Implement error-handling for an unfound dependency (docker build... cannot find the file referenced in FROM something in a dockerfile) #3316
    • Implement error-handling for a docker build failing (docker build... fails for some reason) #3301
    • Implement error-handling for a test not starting properly (docker run... fails for some reason) #3292
    • Clean out prerequisites.sh since it will no longer require most of the packages listed (they will move to the various dockerfiles) #3292
    • Update prerequisites.sh to install Rust and build TFBReaper #3292
    • Update prerequisites.sh to install Docker CE #3292
    • ... Migrate all the shell scripts to dockerfiles
    • Updated scaffolding to reflect the docker implementation (instructions, default files, etc)
    • Update documentation to reflect all these changes
    • Set ulimit on containers (see this)
    • Update --clean implementation to delete all docker images #3327
    • Change database docker build process to only occur when needed as a part of running a test (as opposed to tfb --init) #3312
    • Remove the need for setup_file in all benchmark_config.json and instead just use the name of the test (e.g. gemini.dockerfile for the default, gemini-mysql.dockerfile for gemini-mysql, etc)
    • Implement cli flag --build which requires a test name that will build the docker images but not run anything (useful for testing) #3350
    • Implement environment variable availability for test implementation images; see ARG #3318
    • Change toolset implementation to support multiple FROM in Dockerfiles for staging tools (example: java application but requires maven; create maven.dockerfile to build the webapp in the FROM maven:latest context, then FROM java:latest to run the application) reference #3320
    • Implement docker_files in benchmark_config.json to allow specifying an array of strings which are paths to dockerfiles relative to TROOT which will be considered by the toolset required to be built and run to consider the test 'started' #3321
    • Implement TFB namespacing when creating Docker images #3320
    • Build database images from their primary published docker images (e.g. FROM mysql:5.7 instead of FROM ubuntu:16, etc)
    • Add verification that the proper docker containers are running before the big sleep. #3381
    • Create and publish a docker image suitable for running the toolset (helpful in Travis-CI to go get and run a docker image instead of configuring the environment, installing a bunch of software, etc)
@nbrady-techempower

This comment has been minimized.

Member

nbrady-techempower commented Mar 2, 2018

Sometimes during a test, everything looks like it installed correctly but the docker container(s) for the test are no longer running. Either there was an accidental RUN instead of CMD in the last line of the dockerfile, or the CMD is being run as a daemon when it shouldn't. During the verification process, we should check to make sure all containers from <test-name>.dockerfile and any additional dockerfiles from the benchmark_config.json are still running. Maybe during the 60 second wait for the servers we can do a check at 30 seconds to make sure nothing stopped and give feedback if one of them did. Something like:

INFO:root:Sleeping 60 seconds to ensure framework is ready
INFO:root:Sleeping 30 seconds to ensure framework is ready
ERROR:root Docker container: tfb/test/gemini is not running
@nbrady-techempower

This comment has been minimized.

Member

nbrady-techempower commented Mar 5, 2018

As @greenlaw110 pointed out in a google group post this error wasn't very helpful:

Process Test Runner (act-ebean-pgsql):
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/travis/build/actframework/FrameworkBenchmarks/toolset/benchmark/benchmarker.py", line 608, in __run_test
    result = test.start(out)
  File "/home/travis/build/actframework/FrameworkBenchmarks/toolset/benchmark/framework_test.py", line 228, in start
    deps = list(reversed(gather_docker_dependencies(os.path.join(self.directory, test_docker_file))))
  File "/home/travis/build/actframework/FrameworkBenchmarks/toolset/benchmark/utils.py", line 38, in gather_docker_dependencies
    deps.extend(gather_docker_dependencies(dep_docker_file))
  File "/home/travis/build/actframework/FrameworkBenchmarks/toolset/benchmark/utils.py", line 26, in gather_docker_dependencies
    if os.path.exists(docker_file):
  File "/usr/lib/python2.7/genericpath.py", line 18, in exists
    os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found

It caught me off guard this morning as well, though reading the trace got me to the right place.

I'd like to capture this particular error as well as try to break things in some other common ways and make sure the toolset reports the problem correctly.

@sebastienros

This comment has been minimized.

Contributor

sebastienros commented Mar 8, 2018

Does point 34 mean that you will release tfb/base and other base images on docker hub for instance?

Personally I would like it as it will prevent us from having the build them locally if we want to test the docker images. Saving both time and hassle.

@msmith-techempower

This comment has been minimized.

Member

msmith-techempower commented Mar 8, 2018

@sebastienros Yes, the long-term goal would be that simply running tests involves published images so as to not incur that price of building them all.

@boazsegev

This comment has been minimized.

Contributor

boazsegev commented Mar 25, 2018

I hope this is the right place to post this and you might already be aware of this, but starting vagrant from the docker branch fails to mount the FrameworkBenchmarks folder or install the tbf utility.

(is this a .gitignore related issue?)

The vagrant up call details these errors:

    default: Generating public/private rsa key pair.
    default: Saving key "/home/vagrant/.ssh/id_rsa" failed: No such file or directory
    default: /tmp/vagrant-shell: line 19: /home/vagrant/.ssh/authorized_keys: No such file or directory
    default: chmod: cannot access '/home/vagrant/.ssh/authorized_keys'
    default: : No such file or directory
    default: tee: 
    default: /home/vagrant/.ssh/config
    default: : No such file or directory
    default: NoHostAuthenticationForLocalhost yes
    default: chmod: cannot access '/home/ubuntu/.ssh/config'
    default: : No such file or directory
    default: vagrant ALL=(ALL:ALL) NOPASSWD: ALL
    default: chown: 
    default: invalid user: ‘vagrant:vagrant’
    default: /tmp/vagrant-shell: line 30: cd: /home/ubuntu/FrameworkBenchmarks: No such file or directory
    default: Setting up welcome message
    default: /tmp/vagrant-shell: line 66: ./toolset/setup/linux/prerequisites.sh: No such file or directory
    default: Setting up client and database machines
    default: /tmp/vagrant-shell: line 91: tfb: command not found
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

An attempt to use the utility anyway results in this:

ubuntu@TFB-all:~$ tfb --help
No command 'tfb' found, did you mean:
 Command 'fb' from package 'findbugs' (universe)
 Command 'tf' from package 'tf5' (universe)
 Command 'tf' from package 'tf' (universe)
 Command 'tfm' from package 'emboss' (universe)
 Command 'ttb' from package 'ttb' (universe)
 Command 'tf5' from package 'tf5' (universe)
 Command 'tpb' from package 'tpb' (universe)
 Command 'tnb' from package 'pvm-examples' (universe)
 Command 'thb' from package 'pvm-examples' (universe)
tfb: command not found
ubuntu@TFB-all:~$ cd FrameworkBenchmarks
-bash: cd: FrameworkBenchmarks: No such file or directory
@nbrady-techempower

This comment has been minimized.

Member

nbrady-techempower commented Mar 26, 2018

@boazsegev

not really sure how this line default: /tmp/vagrant-shell: line 30: cd: /home/ubuntu/FrameworkBenchmarks: No such file or directory ever happens with the current set up. I've just gone through the process again and it all worked fine for me. Would you mind destroying that, making sure you have the latest from the docker branch, and trying again. If you get the same results, would you mind adding more info about your host and the versions of vagrant/virtual box

@boazsegev

This comment has been minimized.

Contributor

boazsegev commented Mar 27, 2018

@nbrady-techempower

Solved... I tried rebuilding a number of times to no avail, but than I remembered I have this habit of avoiding updates (I test on older Linux boxes to make sure my code runs on older machines). I updated the Linux box, rebuilt and viola.

Sorry for the false alarm.

B.

@msmith-techempower

This comment has been minimized.

Member

msmith-techempower commented Apr 5, 2018

Closing as #3492 addresses basically everything on the list. We will hodgepodge fixes for the remaining items as we go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment