Docker Extractor (#373)

Author: Or-Geva

README.md

@@ -2,8 +2,7 @@
 
 ## Overview
 
-Build Info is Artifactory's open integration layer for the CI servers and build tools.
-The build information is sent to Artifactory in json format.
+Build Info is Artifactory's open integration layer for the CI servers and build tools. The build information is sent to Artifactory in json format.
 
 ## Building and testing the sources
 
@@ -22,15 +21,15 @@ export BITESTS_ARTIFACTORY_PIP_ENV=/Users/user/venv-test/bin
 ### Building
 When running the following commands to build the code, the entire testing suite is also executed. See the *Testing* section for configuration instructions prior to running the tests.
 
-To build the code using the gradle wrapper in Unix run:  
+To build the code using the gradle wrapper in Unix run:
 ```
 > ./gradlew clean build
 ```
-To build the code using the gradle wrapper in Windows run:  
+To build the code using the gradle wrapper in Windows run:
 ```
 > gradlew clean build
 ```
-To build the code using the environment gradle run:  
+To build the code using the environment gradle run:
 ```
 > gradle clean build
 ```
@@ -40,7 +39,7 @@ To build the code without running the tests, add to the "clean build" command th
 ```
 
 ### Testing
-To run *all* tests:   
+To run *all* tests:
 ```
 > ./gradlew clean test
 ```
@@ -74,8 +73,8 @@ To run *all* tests:
 
 #### Pip tests
 * Add Python and pip executables to the system search path (PATH environment variable).
-* Pip tests must run inside a clean pip-environment. Create a virtual environment and provide its path using the 'BITESTS_ARTIFACTORY_PIP_ENV' variable.  
-When running on a Windows machine, provide the path to the 'Scripts' directory.  
+* Pip tests must run inside a clean pip-environment. Create a virtual environment and provide its path using the 'BITESTS_ARTIFACTORY_PIP_ENV' variable.
+When running on a Windows machine, provide the path to the 'Scripts' directory.
 When running on a unix machine, provide the path to the 'bin' directory.
 ```
 > python -m venv buildinfo-tests-env
@@ -89,6 +88,20 @@ When running on a unix machine, provide the path to the 'bin' directory.
 > ./gradlew clean build-info-extractor-nuget:test
 ```
 
+#### Docker tests
+* Docker tests run only on Linux/mac.
+* In addition to the general environment variables required for running the tests, you must set the following environment variables, required for the docker tests:
+
+ ```
+  export BITESTS_ARTIFACTORY_DOCKER_DOMAIN='server-build-info-tests-docker.jfrog.io/'
+  export BITESTS_ARTIFACTORY_DOCKER_REPO=build-info-tests-docker
+  export BITESTS_ARTIFACTORY_DOCKER_HOST=tcp://127.0.0.1:1234
+ ```
+ ```
+> ./gradlew clean build-info-extractor-docker:test
+```
+* Before running the tests, please make sure you have a local docker repository named *build-info-tests-docker* in Artifactory.
+
 ###  Testing on Artifactory OSS
 When testing with an instance of Artifactory OSS, only supported tests are for the build-info-gradle-extractor.
 
@@ -103,7 +116,7 @@ To run Gradle tests on Artifactory OSS:
 * Create 3 Gradle repositories:
   * Local repository: `build-info-tests-gradle-local`
   * Remote repository to jcenter: `build-info-tests-gradle-remote`
-  * Virtual repository containing both the remote and local: `build-info-tests-gradle-virtual` 
+  * Virtual repository containing both the remote and local: `build-info-tests-gradle-virtual`
 * Run tests `./gradlew build-info-extractor-gradle:test`
 
 ## Build Info json format
@@ -140,7 +153,7 @@ To run Gradle tests on Artifactory OSS:
   }],
   "licenseControl" : {	// Artifactory License Control information
     "runChecks" : true,	// Artifactory will run automatic license scanning after the build is complete (true/false)
-    "includePublishedArtifacts" : true, // Should Artifactory run license checks on the build artifacts, in addition to the build dependecies (true/false) 
+    "includePublishedArtifacts" : true, // Should Artifactory run license checks on the build artifacts, in addition to the build dependecies (true/false)
     "autoDiscover" : true, // Should Artifactory auto discover licenses (true/false)
     "scopesList" : "", // A space-separated list of dependency scopes/configurations to run license violation checks on. If left empty all dependencies from all scopes will be checked.
     "licenseViolationsRecipientsList" : "" // Emails of recipients that should be notified of license violations in the build info (space-separated list)
@@ -151,7 +164,7 @@ To run Gradle tests on Artifactory OSS:
     "minimumBuildDate" : 1407345768020, // Earliest build date to store in Artifactory
     "buildNumbersNotToBeDiscarded" : [ ] // List of build numbers that should not be removed from Artifactory
   },
-  /* List of build modules */	
+  /* List of build modules */
   "modules" : [ { // The build's first module
     "properties" : { // Module properties
       "project.build.sourceEncoding" : "UTF-8"
@@ -182,7 +195,7 @@ To run Gradle tests on Artifactory OSS:
       "project.build.sourceEncoding" : "UTF-8"
     },
     "id" : "org.jfrog.test:multi3:4.2-SNAPSHOT", // Module ID
-    /* List of module artifacts */	
+    /* List of module artifacts */
     "artifacts" : [ { // Module artifacts
       "type" : "war",
       "sha1" : "df8e7d7b94d5ec9db3bfc92e945c7ff4e2391c7c",
@@ -233,8 +246,8 @@ To run Gradle tests on Artifactory OSS:
       "url" : "https://www.jfrog.com/jira/browse/RTFACT-5678",
       "summary" : "Description of the relevant issue",
       "aggregated" : true
-    } ] 
-  },  
+    } ]
+  },
   "governance" : { // Black Duck Code Center integration information
     "blackDuckProperties" : {
       "appName" : "", // The Black Duck Code Center application name

appveyor.yml

@@ -10,6 +10,10 @@ environment:
     secure: Um8o75MQIieSavIemF4ySA==
   BITESTS_ARTIFACTORY_REPO:
     secure: Xf371RQAxCTMn/S7NdmV8g==
+  BITESTS_ARTIFACTORY_DOCKER_DOMAIN:
+    secure: AoaKTKtH3eL36SmENsm51eRFVLU/BGIK+CrBvmsCqO35j3xM9++4E01YWs5CsfXy
+  BITESTS_ARTIFACTORY_DOCKER_REPO:
+    secure: KL72XB/1EBaModiGGpJF9Ge+wdbLmrDwT1lmy28YRiM=
 
   matrix:
     # Extractor Tests
@@ -110,16 +114,21 @@ environment:
       TEST_SUIT:
         "./gradlew clean build-info-extractor-nuget:test"
 
+    # Docker Tests
+    - job_name: docker_extractor_linux
+      job_group: general
+      appveyor_build_worker_image: Ubuntu
+      TEST_SUIT:
+        "./gradlew clean build-info-extractor-docker:test"
+
 for:
-  -
-    matrix:
+  - matrix:
       only:
         - job_group: general
     test_script:
       - sh: "$TEST_SUIT"
       - cmd: "%TEST_SUIT%"
-  -
-    matrix:
+  - matrix:
       only:
         - job_group: pip
     test_script:
@@ -129,4 +138,4 @@ for:
       - cmd: "C:\\Python37-x64\\python -m venv pip-venv"
       - cmd: "%TEST_SUIT%"
 
-build: off
+build: off

build-info-api/src/main/java/org/jfrog/build/api/builder/ArtifactBuilder.java

@@ -19,6 +19,7 @@
 import org.apache.commons.lang.StringUtils;
 import org.jfrog.build.api.Artifact;
 
+import java.util.Map;
 import java.util.Properties;
 
 /**
@@ -164,4 +165,18 @@ public ArtifactBuilder addProperty(Object key, Object value) {
         properties.put(key, value);
         return this;
     }
+
+    /**
+     * Adds properties to the properties object
+     *
+     * @param properties Artifact properties
+     * @return Builder instance
+     */
+    public ArtifactBuilder addProperties(Properties properties) {
+        if (this.properties == null) {
+            this.properties = new Properties();
+        }
+        this.properties.putAll(properties);
+        return this;
+    }
 }

build-info-api/src/main/java/org/jfrog/build/api/builder/DependencyBuilder.java

@@ -19,10 +19,7 @@
 import com.google.common.collect.Lists;
 import org.jfrog.build.api.Dependency;
 
-import java.util.HashSet;
-import java.util.List;
-import java.util.Properties;
-import java.util.Set;
+import java.util.*;
 
 /**
  * A builder for the dependency class
@@ -214,4 +211,18 @@ public DependencyBuilder addProperty(Object key, Object value) {
         properties.put(key, value);
         return this;
     }
+
+    /**
+     * Adds properties to the properties object
+     *
+     * @param properties Dependencies properties
+     * @return Builder instance
+     */
+    public DependencyBuilder addProperties(Properties properties) {
+        if (this.properties == null) {
+            this.properties = new Properties();
+        }
+        this.properties.putAll(properties);
+        return this;
+    }
 }

build-info-extractor-docker/src/main/java/org/jfrog/build/extractor/docker/DockerJavaWrapper.java

@@ -0,0 +1,142 @@
+package org.jfrog.build.extractor.docker;
+
+import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.BuildImageCmd;
+import com.github.dockerjava.api.model.AuthConfig;
+import com.github.dockerjava.core.DefaultDockerClientConfig;
+import com.github.dockerjava.core.DockerClientBuilder;
+import com.github.dockerjava.core.DockerClientConfig;
+import com.github.dockerjava.core.command.BuildImageResultCallback;
+import com.github.dockerjava.core.command.PullImageResultCallback;
+import com.github.dockerjava.core.command.PushImageResultCallback;
+import com.github.dockerjava.netty.NettyDockerCmdExecFactory;
+import org.apache.commons.lang.StringUtils;
+import org.jfrog.build.api.util.Log;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Map;
+
+public class DockerJavaWrapper {
+
+    /**
+     * Push docker image using the docker java client.
+     *
+     * @param imageTag
+     * @param username
+     * @param password
+     * @param host
+     * @param envVars
+     */
+    public static void pushImage(String imageTag, String username, String password, String host, Map<String, String> envVars, Log logger) {
+        final AuthConfig authConfig = new AuthConfig();
+        authConfig.withUsername(username);
+        authConfig.withPassword(password);
+        DockerClient dockerClient = null;
+        try {
+            dockerClient = getDockerClient(host, envVars);
+            dockerClient.pushImageCmd(imageTag).withAuthConfig(authConfig).exec(new PushImageResultCallback()).awaitSuccess();
+        } finally {
+            closeQuietly(dockerClient, logger);
+        }
+    }
+
+    /**
+     * Gets the docker java client.
+     *
+     * @param host    - Docker daemon ip.
+     * @param envVars - System env variables.
+     * @return
+     */
+    public static DockerClient getDockerClient(String host, Map<String, String> envVars) {
+        if (envVars == null) {
+            throw new IllegalStateException("envVars must not be null");
+        }
+        DefaultDockerClientConfig.Builder configBuilder = DefaultDockerClientConfig.createDefaultConfigBuilder();
+        if (envVars.containsKey(DefaultDockerClientConfig.DOCKER_HOST)) {
+            configBuilder.withDockerHost(envVars.get(DefaultDockerClientConfig.DOCKER_HOST));
+        } else {
+            // If open JDK is used and the host is null
+            // then instead of a null reference, the host is the string "null".
+            if (!StringUtils.isEmpty(host) && !host.equalsIgnoreCase("null")) {
+                configBuilder.withDockerHost(host);
+            }
+        }
+        if (envVars.containsKey(DefaultDockerClientConfig.DOCKER_TLS_VERIFY)) {
+            configBuilder.withDockerTlsVerify(envVars.get(DefaultDockerClientConfig.DOCKER_TLS_VERIFY));
+        }
+        if (envVars.containsKey(DefaultDockerClientConfig.DOCKER_CERT_PATH)) {
+            configBuilder.withDockerCertPath(envVars.get(DefaultDockerClientConfig.DOCKER_CERT_PATH));
+        }
+
+        DockerClientConfig config = configBuilder.build();
+        return DockerClientBuilder.getInstance(config).withDockerCmdExecFactory(new NettyDockerCmdExecFactory()).build();
+    }
+
+    /**
+     * Get image Id from imageTag using DockerClient.
+     *
+     * @param imageTag
+     * @param host
+     * @param envVars
+     * @return
+     */
+    public static String getImageIdFromTag(String imageTag, String host, Map<String, String> envVars, Log logger) {
+        DockerClient dockerClient = null;
+        try {
+            dockerClient = getDockerClient(host, envVars);
+            return dockerClient.inspectImageCmd(imageTag).exec().getId();
+        } finally {
+            closeQuietly(dockerClient, logger);
+        }
+    }
+
+    private static void closeQuietly(DockerClient dockerClient, Log logger) {
+        if (dockerClient != null) {
+            try {
+                dockerClient.close();
+            } catch (IOException e) {
+                logger.error("Closes docker client failed.");
+            }
+        }
+    }
+
+    /**
+     * Execute pull docker image on agent.
+     *
+     * @param imageTag
+     * @param username
+     * @param password
+     * @param host
+     * @param envVars
+     */
+    public static void pullImage(final String imageTag, String username, String password, String host, Map<String, String> envVars, Log logger) {
+        final AuthConfig authConfig = new AuthConfig();
+        authConfig.withUsername(username);
+        authConfig.withPassword(password);
+        DockerClient dockerClient = null;
+        try {
+            dockerClient = getDockerClient(host, envVars);
+            dockerClient.pullImageCmd(imageTag).withAuthConfig(authConfig).exec(new PullImageResultCallback()).awaitSuccess();
+        } finally {
+            closeQuietly(dockerClient, logger);
+        }
+    }
+
+    /**
+     * Build a docker image from docker file.
+     *
+     * @param imageName   - The image ID of the result docker build
+     * @param host        - Docker daemon ip.
+     * @param envVars     - System env variables.
+     * @param projectPath - Location of the docker file
+     */
+    public static void buildImage(String imageName, String host, Map<String, String> envVars, String projectPath) {
+        DockerClient dockerClient = DockerJavaWrapper.getDockerClient(host, envVars);
+        // Build the docker image with the name provided from env.
+        BuildImageCmd buildImageCmd = dockerClient.buildImageCmd(Paths.get(projectPath).toFile()).withTags(new HashSet<>(Arrays.asList(imageName)));
+        buildImageCmd.exec(new BuildImageResultCallback()).awaitImageId();
+    }
+}