Docker example update #85

trucemop · 2016-09-25T04:42:08Z

The docker example wasn't working for me in master so I made some tweaks to get it running and added some of the setup steps (cert creations, grafana datasource/dashboard configs, etc) to the scripts to decrease manual config time.

Steps should now be:
mvn package
cd ./server;docker-compose up
login to grafana and click browser-cert check in the timely datasource.

ctubbsii

I don't know much about docker-specific issues, but I've added some comments regarding minor generic packaging, dependency, and scripting issues.

ctubbsii · 2016-09-25T07:41:59Z

server/Dockerfile

-  mkdir -p /timely/bin
+  yum -y install java-1.8.0-openjdk-devel maven openssl apr wget && \
+  mkdir -p /timely/bin && \
+  wget -O /usr/lib/jvm/jre/lib/ext/bcprov-jdk15on-155.jar https://www.bouncycastle.org/download/bcprov-jdk15on-155.jar && \


Why not just add EPEL repo and do yum install bouncycastle?

Also, the standard location for jars is in /usr/share/java, with symlinks pointing to this location if necessary. That way, the /usr/bin/build-classpath tool works properly.

Added EPEL, installed via yum, and created symlink.

If there's a base image out there with this already in it, I'd prefer to use that

ctubbsii · 2016-09-25T07:44:45Z

server/Dockerfile

+  yum -y install java-1.8.0-openjdk-devel maven openssl apr wget && \
+  mkdir -p /timely/bin && \
+  wget -O /usr/lib/jvm/jre/lib/ext/bcprov-jdk15on-155.jar https://www.bouncycastle.org/download/bcprov-jdk15on-155.jar && \
+  echo "security.provider.9=org.bouncycastle.jce.provider.BouncyCastleProvider" >> /usr/lib/jvm/jre/lib/security/java.security


Additional, user-configured providers should go in /etc/java/security/security.d. /usr doesn't typically hold user-edited configs.

/etc/java/security/security.d/2000-org.bouncycastle.jce.provider.BouncyCastleProvider is created as part of the bouncy castle install, but timely still throws unless I add the provider to java.security. I only did it this way originally because it is how the bouncy castle wiki says to add statically. I'm noob at these bits so just following instructions I find.

ctubbsii · 2016-09-25T07:50:49Z

server/Dockerfile


-ENV JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el7_2.x86_64
+ENV JAVA_HOME /usr/lib/jvm/java


If Java 8 is specifically desired, it'd be better to use: /usr/lib/jvm/java-1.8.0-openjdk

I'm also partial to using $(dirname $(dirname $(readlink -e $(which java)))) (or javac, if JDK is required for JAVA_HOME)

Changed JAVA_HOME to /usr/lib/jvm/java-1.8.0-openjdk. I might not be Dockerfile savvy enough, but subshell in the ENV statement seemed to be problematic.

ctubbsii · 2016-09-25T07:51:52Z

server/bin/docker/deploy-startup-standalone.sh

@@ -1,6 +1,11 @@
 #!/bin/sh

+WORKING="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+. ${WORKING}/deploy-env.sh


paths should be quoted when used, in case they have spaces
I recommend passing script through shellcheck (shellcheck.net or CLI version)

Ran through shellcheck. Quoted dirs.

ctubbsii · 2016-09-25T07:52:33Z

server/bin/docker/deploy-startup-standalone.sh

-/timely/bin/timely-standalone.sh
+${SRC_DIR}/bin/docker/genCerts.sh
+${SRC_DIR}/bin/docker/deploy-timely-build.sh
+${TIMELY_DIR}/bin/timely-standalone.sh


More unquoted path variables. Lots more in other scripts, but I've stopped noting them.

Ran through shellcheck. Quoted dirs.

ctubbsii · 2016-09-25T07:55:33Z

server/bin/docker/genCerts.sh

+
+	#Create a private key
+
+	openssl genrsa -des3 -out CA.key -passout ${PASSWORD} 4096


password variable should be quoted (suggest passing whole script through shellcheck)

Also, it's generally safer to use the env: prefix for passwords, and put the password in the environment. That way, it's not visible to other users who can list processes.

It appears that this is copied from our documentation. We should update that too.

Ran through shellcheck. Quoted dirs. Switched to env:PASSWORD

ctubbsii · 2016-09-25T08:05:47Z

server/bin/docker/genCerts.sh

+
+	#Create a private key
+
+	openssl genrsa -des3 -out CA.key -passout ${PASSWORD} 4096


Why not use -aes256 instead of -des3?

Switched to -aes256. Was just following the steps in the timely doc, which was using the steps in the accumulo doc.

…uotes, env: prefix for pass.

klucar · 2016-09-26T00:14:38Z

In general, this was more of a way for me to develop the Grafana plugin against and I modify it to handle whatever I'm working on. You can already create your own certs and mount them in the image, which is what I do when I want to run with certs. #58 is there to generate an official docker image capable of handling production use.

Perhaps I should just move this out into a repo of my own and we can point the docs to an official Docker image.

trucemop · 2016-09-26T00:50:05Z

The main goal of this PR was to have a working thing with fewer steps than were currently required. When I first tried (and failed) to get timely running I was happy to see the Dockerfile and docker-compose.yml containing all of the bits necessary to get started. I saw that it was already configured to use whatever was compiled, thought that was pretty useful, and thought it would be neat if docker-compose up was all that was necessary after compilation.

I saw #58 and having the official image would be awesome. I also thought that having these bits might still be handy for development.

veteranbv · 2016-09-26T00:55:37Z

+1 I was able to get the example running much more easily using this, thanks @trucemop

klucar · 2016-09-26T13:28:23Z

server/bin/docker/deploy-wait-and-insert.sh

+# star the dashboard
+curl -k -X POST https://admin:admin@grafana:3000/api/user/stars/dashboard/1
+
+"${TIMELY_DIR}/bin/insert-test-data.sh"


The timely grafana app has a standalone dashboard in it now. Is this post importing that one or uploading a separate one?

The curls I added:

-Enable the app
-Configure the datasource
-Import the existing dashboard
-Star the dashboard (just to highlight it a bit more)

The docker-compose wasn't modified much. It still adds local grafana/timely-app as a volume mounted to the grafana container's /var/lib/grafana/plugins/timely-app

klucar · 2016-09-26T13:31:06Z

server/bin/docker/standalone_test.json

@@ -173,7 +173,7 @@
              "downsampleAggregator": "max",
              "downsampleFillPolicy": "none",
              "downsampleInterval": "1s",
-              "metric": "sys.cpu0.user",
+              "metric": "sys.cpu.user",


This has all been fixed and added as a standard dashboard to the timely app. see grafana/timely-app/src/datasource/dashboards/standalone_test.json

After I got the docker bits working I noticed that sys.cpu0.user wasn't being inserted by timely.util.InsertTestData so I just changed all of the standalone_test.json sys.cpu0.user to sys.cpu.user where I found it under the assumption they were both intended to be used together.

I didn't change the volume use in the docker-compose so it still uses grafana/timely-app/src/datasource/dashboards/standalone_test.json.

I just saw that I haven't pushed this fix from my local box yet (you're welcome) Anyways, this file should be deleted and these changes should be made to grafana/timely-app/src/datasource/dashboards/standalone_test.json The curl POST you added imports that file, not this one.

Removed the file. I already made the changes to grafana/timely-app/src/datasource/dashboards/standalone_test.json

klucar · 2016-10-04T20:21:33Z

I pulled this branch and ran into a few issues.

The timely app is enabled but the menu doesn't appear under the grafana menu/sidebar. This prevents people from logging in to the datasource and viewing the available metrics.

Also, queries produced this error:
java.lang.IllegalArgumentException: Cannot support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 with currently installed providers

trucemop · 2016-10-04T22:08:26Z

Cannot support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 is the exception I was getting before I included the bouncycastle jar and added it as a security provider. Are there existing images on your box getting run or timely builds that are getting used by the volume config/standalone deploy scripts?

I tested the bits for this by checking out to a new location and cleaning all of my existing docker containers/images on fedora and then later a fresh checkout->build->"docker-compose build" on a macos box that had never built timely and needed docker installed.

I definitely ran into some weirdness depending on the state of the images and timely build. I will check stuff out again tonight.

klucar · 2016-10-04T23:44:39Z

Ok, I'll wipe my images and give it another go.

On Tue, Oct 4, 2016 at 6:08 PM, John Highcock notifications@github.com
wrote:

Cannot support TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 is the exception I
was getting before I included the bouncycastle jar and added it as a
security provider. Are there existing images on your box getting run or
timely builds that are getting used by the volume config/standalone deploy
scripts?

I tested the bits for this by checking out to a new location and cleaning
all of my existing docker containers/images on fedora and then later a
fresh checkout->build->"docker-compose build" on a macos box that had
never built timely and needed docker installed.

I definitely ran into some weirdness depending on the state of the images
and timely build. I will check stuff out again tonight.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#85 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAmkIIr4Mnflg4JG3w9owMQUk5-U6Qg2ks5qws5agaJpZM4KF0UF
.

trucemop · 2016-10-05T02:40:00Z

Just got a chance to validate again. Using the below steps I was able to login to grafana, select data sources, select timely, and click browser cert check. The timely dashboard was populated with data at that point. I realize the docker cleaning is a bit heavy handed, but wanted to make sure I wasn't using any existing image contents.

git clone https://github.com/trucemop/timely.git
cd timely
git checkout docker
mvn clean package
docker ps -a | awk '{print $1}' | grep -v CONTAINER | xargs docker rm -f
docker images -a | awk '{print $3}' | grep -v IMAGE | xargs docker rmi -f
cd server
docker-compose up

klucar · 2016-10-05T10:32:18Z

Its still not working for me. Timely isn't showing up in the Grafana menu still and https://localhost:54322/api/metrics returns "ResponseCode: 404 Message: Returning HTTP Strict Transport Security response"

klucar · 2016-10-05T11:08:08Z

Found two issues. Needed to do this:

export PASSWORD=blargh

Also, "pinned":true to the enable app curl makes Timely appear in the menu

# enable app
curl -k -H "Content-Type: application/json" -d \
'{
  "enabled":true,
  "pinned":true
}' \
 https://admin:admin@grafana:3000/api/plugins/timely-app/settings

However, I have to go into the datasource and click test&save to get the datasources visible in the timely app pages (Login/Metrics) My guess is there's a parameter missing from the create datasource post. I'll look into it more later.

klucar · 2016-10-11T00:25:24Z

I merged this, then added my fix and have been working with it for the accumulo summit talk, so it seems to be going fine.

trucemop added 6 commits September 16, 2016 15:55

Remove TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 exception in Dockerfile.

ec1ac9a

Working again.

a238943

Can now just run 'docker-compose up'

12e6431

Add the grafana bits so that datasource and dashboard are setup.

a4b9a10

Merge branch 'master' into docker

b28cb44

Star the timely dashboard

c8646a8

ctubbsii requested changes Sep 25, 2016

View reviewed changes

ctubbsii reviewed Sep 25, 2016

View reviewed changes

Addressing some of the comments. JAVA_HOME specific version, script q…

9c75c75

…uotes, env: prefix for pass.

dlmarion assigned klucar Sep 25, 2016

Install bouncycastle via epel-release

a034223

klucar reviewed Sep 26, 2016

View reviewed changes

Remove extra standalone_test.json

186edeb

klucar merged commit 8f46ee0 into NationalSecurityAgency:master Oct 10, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker example update #85

Docker example update #85

trucemop commented Sep 25, 2016

ctubbsii left a comment

ctubbsii Sep 25, 2016 •

edited

trucemop Sep 25, 2016

klucar Sep 26, 2016

ctubbsii Sep 25, 2016

trucemop Sep 25, 2016

ctubbsii Sep 25, 2016

trucemop Sep 25, 2016

ctubbsii Sep 25, 2016

trucemop Sep 25, 2016

ctubbsii Sep 25, 2016 •

edited

trucemop Sep 25, 2016

ctubbsii Sep 25, 2016 •

edited

dlmarion Sep 25, 2016

trucemop Sep 25, 2016

ctubbsii Sep 25, 2016

trucemop Sep 25, 2016

klucar commented Sep 26, 2016

trucemop commented Sep 26, 2016

veteranbv commented Sep 26, 2016

klucar Sep 26, 2016

trucemop Sep 26, 2016

klucar Sep 26, 2016

trucemop Sep 26, 2016

klucar Sep 26, 2016

trucemop Sep 26, 2016

klucar commented Oct 4, 2016

trucemop commented Oct 4, 2016

klucar commented Oct 4, 2016

trucemop commented Oct 5, 2016

klucar commented Oct 5, 2016

klucar commented Oct 5, 2016

klucar commented Oct 11, 2016


		ENV JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.91-1.b14.el7_2.x86_64
		ENV JAVA_HOME /usr/lib/jvm/java


		#Create a private key

		openssl genrsa -des3 -out CA.key -passout ${PASSWORD} 4096

Docker example update #85

Docker example update #85

Conversation

trucemop commented Sep 25, 2016

ctubbsii left a comment

Choose a reason for hiding this comment

ctubbsii Sep 25, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctubbsii Sep 25, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ctubbsii Sep 25, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

klucar commented Sep 26, 2016

trucemop commented Sep 26, 2016

veteranbv commented Sep 26, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

klucar commented Oct 4, 2016

trucemop commented Oct 4, 2016

klucar commented Oct 4, 2016

trucemop commented Oct 5, 2016

klucar commented Oct 5, 2016

klucar commented Oct 5, 2016

klucar commented Oct 11, 2016

ctubbsii Sep 25, 2016 •

edited

ctubbsii Sep 25, 2016 •

edited

ctubbsii Sep 25, 2016 •

edited