Skip to content

Commit

Permalink
Upgraded clj-docker-client to 0.5.2
Browse files Browse the repository at this point in the history 
 - Refactored the interfacing functions to use data-driven client
 - Added different connections for different tasks to multiplex
 - Amended tests to adapt to new client functions
 - Refactored some naming to be more precise
 - Closes bob-cd#68
 - Closes bob-cd#56
 - Refers bob-cd#20 into-docker/clj-docker-client#20

 Most of the time it was sufficient to just change the function call
 to run docker/invoke with a map of options. The new client returns
 a message when there was a problem, so I needed to catch that to
 return a failjure. Tests had to be changed to keep up with the
 changed function calls.

 A new connection is made everytime invoke is called with the new
 clj-docker-client. The url bob connects to is now configured via env-var.
 That solves the problem we had with the former version of the client-lib.
 Documentation shows handling of variables set globally.

Squashed commit of the following:

commit e6b5a8bad70bbe331dcebbf1cc4fa053bbc6167e
Author: Timo Kramer <fw-git@timokramer.de>
Date:   Tue Mar 17 20:36:36 2020 +0100

    using patched docker-lib to solve socket-problem

commit 68b967459f9100f022b924cc5242dce09280c7d9
Author: Timo Kramer <fw-git@timokramer.de>
Date:   Tue Mar 17 15:57:32 2020 +0100

    Revert "trying different things to solve socket problems"

    This reverts commit 57a335e.

commit 57a335e
Author: Timo Kramer <fw-git@timokramer.de>
Date:   Sun Mar 8 15:36:52 2020 +0100

    trying different things to solve socket problems

commit 21ad5f9
Author: Timo Kramer <fw-git@timokramer.de>
Date:   Fri Mar 6 13:15:51 2020 +0100

    Upgraded clj-docker-client to 0.5.1

    - Refactored the interfacing functions to use data-driven client
    - Added different connections for different tasks to multiplex
    - Amended tests to adapt to new client functions
    - Refactored some naming to be more precise
    - Closes bob-cd#68

    Most of the time it was sufficient to just change the function call
    to run docker/invoke with a map of options. The new client returns
    a message when there was a problem, so I needed to catch that to
    return a failjure. Tests had to be changed to keep up with the
    changed function calls.

    Still there is a multiplex-problem when pulling multiple images.

    Squashed commit of the following:

    commit 64b6a32
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Thu Mar 5 20:23:18 2020 +0100

        improved some tests

    commit 554fa11
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Mar 3 15:06:08 2020 +0100

        the errors were not printed correctly

    commit 0915bfe
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Mar 3 15:05:45 2020 +0100

        extra connection for a pull because experiencing Socket problems

    commit ab297d4
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Mar 3 01:02:04 2020 +0100

        less delay because pipeline is passed and cannot be stopped

    commit a6c5c3a
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Mar 2 23:32:13 2020 +0100

        logging was not working because of special chars

    commit 5fd3b69
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Mar 2 14:15:12 2020 +0100

        TODO was superfluous

    commit 47fc68a
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Mar 2 12:24:55 2020 +0100

        refactored run to start-container because more accurate

    commit 04d51c0
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Mar 2 12:14:33 2020 +0100

        cider-formatting

    commit 50ea2fc
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Mar 2 12:11:08 2020 +0100

        refactored function build to create-container because that is more accurate

    commit aec1d81
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Mar 2 11:34:15 2020 +0100

        comment corrected

    commit f7eb578
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Mar 2 10:58:07 2020 +0100

        stream to artifact-store works now

    commit 509cbc9
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Fri Feb 28 18:32:24 2020 +0100

        bootstrap_wendy.sh fails on extracting the tar

    commit aed0816
    Merge: 05e1d75 afa375b
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Fri Feb 28 14:50:09 2020 +0100

        Merge remote-tracking branch 'bob/master' into bob-cd#68-upgrade-docker-client

    commit 05e1d75
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Wed Feb 26 16:24:10 2020 +0100

        refactored

    commit e939327
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Wed Feb 26 10:15:39 2020 +0100

        mistake in health-check was always returning unhealthy or errored out on connection issue

    commit f9577df
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Feb 25 15:38:05 2020 +0100

        fixed malformed return of commit-image function

    commit 9d05e0d
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Feb 25 12:41:14 2020 +0100

        took functions to format CMD and ENV for passing to daemon

    commit 1217e31
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Feb 24 18:50:08 2020 +0100

        TODO added

    commit 5c33c05
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Feb 24 14:05:13 2020 +0100

        log-streaming implemented

    commit bef9118
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Sun Feb 23 15:26:17 2020 +0100

        updated docker-client

    commit 1bfddf1
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Sun Feb 23 15:26:03 2020 +0100

        timeouts added for docker connection

    commit 180262a
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Sun Feb 23 15:25:42 2020 +0100

        bug filed for clj-docker on timeout on putcontainerarchive

    commit 4298a4f
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Feb 18 17:15:36 2020 +0100

        garbage collection repaired

    commit 2036b96
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Feb 18 17:15:12 2020 +0100

        refactored execution internals

    commit 5b3b25b
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Feb 18 14:06:07 2020 +0100

        fixed some tests

    commit a550949
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Feb 18 11:36:02 2020 +0100

        pipeline internals adapted

    commit fbf8a98
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Feb 17 17:55:39 2020 +0100

        refactored commit-image into execution/internals and adapted pipeline/internals next-step to new clj-docker-client

    commit e7084b3
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Sat Feb 15 18:21:15 2020 +0100

        failing test; dont know how to mock file

    commit 44acdb9
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Sat Feb 15 18:20:20 2020 +0100

        made commit a state

    commit d9e003f
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Fri Feb 14 16:06:23 2020 +0100

        resource internals adapted

    commit 74baf38
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Feb 10 16:01:08 2020 +0100

        artifact/core adapted

    commit 6b81cce
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Mon Feb 10 12:44:52 2020 +0100

        health check adapted

    commit 5c9f70b
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Thu Feb 6 17:50:31 2020 +0100

        run function using new clj-docker

    commit 448df78
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Thu Feb 6 16:56:54 2020 +0100

        improved build to use failjure return

    commit 1f89b75
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Thu Feb 6 16:22:16 2020 +0100

        status-of function now using new clj-docker lib

    commit 8de9273
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Thu Feb 6 15:06:03 2020 +0100

        validation for image name improved and container build implemented

    commit b8eda9d
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Tue Feb 4 15:01:02 2020 +0100

        log-and-fail to avoid repetition

    commit d38ff6c
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Sun Feb 2 18:36:38 2020 +0100

        pull image function refactored

    commit c7a4c66
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Thu Jan 30 16:05:27 2020 +0100

        switched to states

    commit 6d51bd2
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Wed Jan 29 13:16:42 2020 +0100

        kill-container

    commit d944ebc
    Author: Timo Kramer <fw-git@timokramer.de>
    Date:   Wed Jan 29 10:51:55 2020 +0100

        first function transitioned to clj-docker 0.4.0
  • Loading branch information
@TimoKramer
TimoKramer committed Mar 23, 2020
1 parent afa375b commit c8bb741
Show file tree
Hide file tree
Showing 16 changed files with 765 additions and 356 deletions.
17 changes: 16 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,24 @@ See the Kanban [board](https://github.com/bob-cd/bob/projects/1) to see the road

## Running requirements
- Any OS supporting Java and Docker
- JDK 8 to 12. 13+ will be supported when [#56](https://github.com/bob-cd/bob/issues/56) is fixed.
- JDK 8 to 13
- Docker (latest preferred for optimal performance)

## Configuration
Configuration uses the [environ library](https://github.com/weavejester/environ) and therefore several variables can be
set by specifying them as environment variable or as java system property. Possible variables are:
| java system properties | environment variables | defaults |
|------------------------|-----------------------|-----------------------------|
| bob-db-host | BOB_DB_HOST | localhost |
| bob-db-port | BOB_DB_PORT | 5432 |
| bob-db-user | BOB_DB_USER | bob |
| bob-db-name | BOB_DB_NAME | bob |
| bob-docker-uri | BOB_DOCKER_URI | unix:///var/run/docker.sock |
| bob-connect-timeout | BOB_CONNECT_TIMEOUT | 1000ms |
| bob-read-timeout | BOB_READ_TIMEOUT | 30000ms |
| bob-write-timeout | BOB_WRITE_TIMEOUT | 30000ms |
| bob-call-timeout | BOB_CALL_TIMEOUT | 40000ms |

## Testing, building and running locally
- Clone this repository.
- Install the Build requirements.
Expand Down
2 changes: 1 addition & 1 deletion build.boot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
[com.layerware/hugsql "0.5.1"]
[metosin/compojure-api "2.0.0-alpha30"]
[prismatic/schema "1.1.12"]
[lispyclouds/clj-docker-client "0.3.2"]
[lispyclouds/clj-docker-client "0.5.2"]
[mount "0.1.16"]
[environ "1.1.0"]
[com.impossibl.pgjdbc-ng/pgjdbc-ng "0.8.3"]
Expand Down
15 changes: 13 additions & 2 deletions integration-tests/bob-tests.strest.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -213,11 +213,22 @@ requests:
- jsonpath: "content.message"
expect: "Ok"

secondPipelineStatusRunning:
request:
url: "http://bob:7777/api/pipelines/status/groups/dev/names/test1/number/1"
method: "GET"
delay: 2000
validate:
- jsonpath: "status"
expect: 200
- jsonpath: "content.message"
expect: "running"

pipelineStop:
request:
url: "http://bob:7777/api/pipelines/stop/groups/dev/names/test1/number/1"
method: "POST"
delay: 10000
delay: 2000
validate:
- jsonpath: "status"
expect: 200
Expand Down Expand Up @@ -299,7 +310,7 @@ requests:
- jsonpath: "content.message"
expect: "passed"

secondPipelineStatus:
secondPipelineStatusStopped:
request:
url: "http://bob:7777/api/pipelines/status/groups/dev/names/test1/number/1"
method: "GET"
Expand Down
28 changes: 18 additions & 10 deletions src/bob/api/health.clj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,21 +41,23 @@
"Check the systems we depend upon. Returns nil if everything is alright, else returns a sequence
of strings naming the failing systems."
[]
(let [docker (when (f/failed? (f/try* (docker/ping states/docker-conn)))
["Docker"])
postgres (when (f/failed? (f/try* (db-health-check states/db)))
["Postgres"])
extsys (when (nil? postgres)
(ping-external-systems))]
(let [docker (when (not (= "OK" (docker/invoke {:category :_ping
:conn states/conn}
{:op :SystemPing})))
["Docker"])
postgres (when (f/failed? (f/try* (db-health-check states/db)))
["Postgres"])
extsys (when (nil? postgres)
(ping-external-systems))]
(filter some? (concat docker postgres extsys))))

(defn respond-to-health-check
"Endpoint for answering a health check"
[]
(d/let-flow [failures (health-check)]
(if (empty? failures)
(u/respond "Yes we can! \uD83D\uDD28 \uD83D\uDD28")
(u/service-unavailable (str "Health check failed: " (clojure.string/join " and " failures) " not healthy")))))
(if (empty? failures)
(u/respond "Yes we can! \uD83D\uDD28 \uD83D\uDD28")
(u/service-unavailable (str "Health check failed: " (clojure.string/join " and " failures) " not healthy")))))

(defn log-health-check
"Logs if any of the subsystems is unhealthy."
Expand Down Expand Up @@ -83,14 +85,20 @@
(a/close! heartbeat))

(comment
(docker/categories)
(def ping (docker/client {:category :_ping :conn states/conn}))
(docker/ops (docker/client {:category :_ping :conn states/conn}))
(docker/invoke ping {:op :SystemPing})
(f/failed? (f/try* (docker/invoke ping {:op :SystemPing})))

(when (nil? nil) (concat (get-artifact-stores states/db) (get-external-resources states/db)))

(map #(when (f/failed? (f/try* @(http/get (clojure.string/join "/" [(:url %) "ping"]) {:throw-exceptions false}))) (:name %))
(concat (get-artifact-stores states/db) (get-external-resources states/db)))

(ping-external-systems)

(str "Health check failed: " (clojure.string/join " and " (health-check)) " not healthy")
(health-check)

(log-health-check)

Expand Down
20 changes: 17 additions & 3 deletions src/bob/artifact/core.clj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
[taoensso.timbre :as log]
[bob.util :as u]
[bob.states :as states]
[bob.execution.internals :as int]
[bob.artifact.db :as db]))

(defn register-artifact-store
Expand Down Expand Up @@ -95,7 +96,11 @@
Returns a Failure object if failed."
[group name number artifact run-id path store-name]
(if-let [{url :url} (db/get-artifact-store states/db {:name store-name})]
(f/try-all [stream (docker/stream-path states/docker-conn run-id path)
(f/try-all [_ (log/debugf "Streaming from container with id %s and path %s" run-id path)
stream (docker/invoke states/containers {:op :ContainerArchive
:params {:id run-id
:path path}
:as :stream})
upload-url (clojure.string/join "/"
[url
"bob_artifact"
Expand All @@ -114,13 +119,22 @@
"Ok"
(f/when-failed [err]
(log/errorf "Error in uploading artifact: %s" (f/message err))
(f/try* (docker/rm states/docker-conn run-id))
(f/try* (int/delete-container run-id :force))
err))
(do (log/error "Error locating Artifact Store")
(f/try* (docker/rm states/docker-conn run-id))
(f/try* (int/delete-container run-id :force))
(f/fail "No such artifact store registered"))))

(comment
(docker/ops (docker/client {:category :containers :conn states/conn}))
(let [foo (docker/invoke states/containers {:op :ContainerArchive
:params
{:id "clever_swartz"
:path "/root/my-source/target/default+uberjar/wendy"}
:as :stream})]
@(http/post "http://localhost:8001/bob_artifact/wendy/build/3/wendy" {:multipart [{:name "data" :content foo}]}))
(upload-artifact "dev" "test" 1 "poetry.lock" "0be7b883382b" "/opt" "s3")

(db/register-artifact-store states/db {:name "s3"
:url "http://localhost:8001"})

Expand Down
Loading

0 comments on commit c8bb741

Please sign in to comment.