feat (hatchery/swarm): --add-host options

docs/content/hatchery/swarm.md

@@ -30,4 +30,4 @@ This hatchery will now start worker of model 'docker' on you docker installation
 
 ## Setup a worker model
 
-See [Tutorial]({{< relref "workflows/pipelines/requirements/worker-model/docker-simple.md" >}})
+See [Tutorial]({{< relref "workflows/pipelines/requirements/worker-model/docker/_index.md" >}})

docs/content/hosting/ready-to-run/docker-compose.md

@@ -102,7 +102,7 @@ $ docker-compose up cds-hatchery-swarm
 A `swarm hatchery` spawns CDS Workers inside dedicated containers.
 This ensures isolation of the workspaces and resources.
 
-Now, you have to create worker model of type `docker`, please follow [how to create a worker model docker]({{< relref "workflows/pipelines/requirements/worker-model/docker-simple.md" >}}).
+Now, you have to create worker model of type `docker`, please follow [how to create a worker model docker]({{< relref "workflows/pipelines/requirements/worker-model/docker/_index.md" >}}).
 
 ## Next with Actions, Plugins
 

docs/content/workflows/pipelines/requirements/service/example-pg.md

@@ -18,7 +18,7 @@ POSTGRES_USER=myuser
 POSTGRES_PASSWORD=mypassword
 ```
 
-And a requirement model which allow you to execute `apt-get install -y postgresql-client`, see [HowTo]({{< relref "workflows/pipelines/requirements/worker-model/docker-simple.md" >}})
+And a requirement model which allow you to execute `apt-get install -y postgresql-client`, see [HowTo]({{< relref "workflows/pipelines/requirements/worker-model/docker/_index.md" >}})
 
 
 ![Requirement](/images/tutorials_service_link_pg_requirements.png)

docs/content/workflows/pipelines/requirements/worker-model/_index.md

@@ -15,7 +15,7 @@ The goal of a worker model is to describe the capabilities of a given docker/iso
 
 There are 2 types of worker models:
 
- * Docker images, see [how to create a worker model docker]({{< relref "workflows/pipelines/requirements/worker-model/docker-simple.md" >}})
+ * Docker images, see [how to create a worker model docker]({{< relref "workflows/pipelines/requirements/worker-model/docker/_index.md" >}})
  * Openstack images, see [how to create a worker model openstack]({{< relref "workflows/pipelines/requirements/worker-model/openstack.md" >}})
 
 ### Behavior

docs/content/workflows/pipelines/requirements/worker-model/docker/_index.md

@@ -0,0 +1,36 @@
++++
+title = "Docker Worker Model"
+weight = 1
+
++++
+
+A worker model of type `docker` can be spawned by a Hatchery Docker Swarm or a Hatchery Marathon.
+
+## Register a worker Model from an existing Docker Image
+
+Docker Image *golang:1.8.1* have a "curl" in $PATH, so it can be used as it is.
+
+* In the UI, click on the wheel on the hand right top corner and select *workers" (or go the the route *#/worker*)
+* At the bottom of the page, fill the form
+    * Name of your worker *Golang-1.8.1*
+    * type *docker*
+    * image *golang:1.8.1*
+* Click on *Add* button and that's it
+
+![Add worker model](/images/workflows.pipelines.requirements.docker.worker-model.docker.add.png)
+
+## Worker Model Docker on Hatchery Swarm
+
+This hatchery offers some features on job pre-requisites, usable only on user's hatchery (ie. not a shared.infra hatchery).
+
+* [Service Link]({{< relref "workflows/pipelines/requirements/service/_index.md" >}})
+* options on worker model prerequisite
+    * Port mapping: `--port=8080:8081/tcp --port=9080:9081/tcp`
+    * Priviledge flag: `--privileged`
+    * Add host flag: `--add-host=aaa:1.2.3.4 --add-host=bbb:5.6.7.8`
+    * Use all: `--port=8080:8081/tcp --privileged --port=9080:9081/tcp --add-host=aaa:1.2.3.4 --add-host=bbb:5.6.7.8`
+* options on volume prerequisite
+    * Bind: `type=bind,source=/hostDir/sourceDir,destination=/dirInJob,readonly`
+
+![Job Prerequisites](/images/workflows.pipelines.requirements.docker.worker-model.docker.png)
+

engine/hatchery/swarm/swarm.go

@@ -458,6 +458,11 @@ func (h *HatcherySwarm) listAwolWorkers() ([]types.Container, error) {
 	//Checking workers
 	oldContainers := []types.Container{}
 	for _, c := range containers {
+		if time.Now().Add(-1*time.Minute).Unix() < c.Created {
+			log.Debug("listAwolWorkers> container %s is too young", c.Names[0])
+			continue
+		}
+
 		//If there isn't any worker registered on the API. Kill the container
 		if len(apiworkers) == 0 {
 			oldContainers = append(oldContainers, c)
@@ -479,10 +484,12 @@ func (h *HatcherySwarm) listAwolWorkers() ([]types.Container, error) {
 		}
 		//If the container doesn't match any worker : Kill it.
 		if !found {
+			log.Debug("listAwolWorkers> %s not found, add to oldContainers", c.Names[0])
 			oldContainers = append(oldContainers, c)
 		}
 	}
 
+	log.Debug("listAwolWorkers> oldContainers: %d", len(oldContainers))
 	return oldContainers, nil
 }
 
@@ -514,18 +521,14 @@ func (h *HatcherySwarm) killAwolWorker() error {
 		}
 		//check if the service is linked to a worker which doesn't exist
 		if w, _ := h.getContainer(c.Labels["service_worker"], types.ContainerListOptions{All: true}); w == nil {
-			oldContainers = append(oldContainers, c)
+			log.Debug("killAwolWorker> Delete worker (service) %s", c.Names[0])
+			if err := h.killAndRemove(c.ID); err != nil {
+				log.Error("killAwolWorker> service %v", err)
+			}
 			continue
 		}
 	}
 
-	for _, c := range oldContainers {
-		log.Debug("killAwolWorker> Delete worker %s", c.Names[0])
-		if err := h.killAndRemove(c.ID); err != nil {
-			log.Error("killAwolWorker> %v", err)
-		}
-	}
-
 	return h.killAwolNetworks()
 }
 

engine/hatchery/swarm/swarm_util_create.go

@@ -53,7 +53,6 @@ func (h *HatcherySwarm) createAndStartContainer(cArgs containerArgs) error {
 	log.Debug("createAndStartContainer> Create container %s from %s on network %s as %s (memory=%dMB)", cArgs.name, cArgs.image, cArgs.network, cArgs.networkAlias, cArgs.memory)
 
 	var exposedPorts nat.PortSet
-	var mounts []mount.Mount
 
 	name := cArgs.name
 	config := &container.Config{
@@ -71,7 +70,8 @@ func (h *HatcherySwarm) createAndStartContainer(cArgs containerArgs) error {
 	hostConfig := &container.HostConfig{
 		PortBindings: cArgs.dockerOpts.ports,
 		Privileged:   cArgs.dockerOpts.privileged,
-		Mounts:       mounts,
+		Mounts:       cArgs.dockerOpts.mounts,
+		ExtraHosts:   cArgs.dockerOpts.extraHosts,
 	}
 	hostConfig.Resources = container.Resources{
 		Memory:     cArgs.memory * 1024 * 1024, //from MB to B
@@ -105,6 +105,7 @@ type dockerOpts struct {
 	ports      nat.PortMap
 	privileged bool
 	mounts     []mount.Mount
+	extraHosts []string
 }
 
 func computeDockerOpts(isSharedInfra bool, requirements []sdk.Requirement) (*dockerOpts, error) {
@@ -140,6 +141,10 @@ func (d *dockerOpts) computeDockerOptsOnModelRequirement(isSharedInfra bool, req
 			if err := d.computeDockerOptsPorts(opt); err != nil {
 				return err
 			}
+		} else if strings.HasPrefix(opt, "--add-host=") {
+			if err := d.computeDockerOptsExtraHosts(opt); err != nil {
+				return err
+			}
 		} else if opt == "--privileged" {
 			d.privileged = true
 		} else {
@@ -215,6 +220,12 @@ func (d *dockerOpts) computeDockerOptsOnVolumeMountRequirement(opt string) error
 	return nil
 }
 
+func (d *dockerOpts) computeDockerOptsExtraHosts(arg string) error {
+	value := strings.TrimPrefix(strings.TrimSpace(arg), "--add-host=")
+	d.extraHosts = append(d.extraHosts, value)
+	return nil
+}
+
 func (d *dockerOpts) computeDockerOptsPorts(arg string) error {
 	if regexPort.MatchString(arg) {
 		s := regexPort.FindStringSubmatch(arg)

engine/hatchery/swarm/swarm_util_create_test.go

@@ -103,6 +103,19 @@ func Test_computeDockerOpts(t *testing.T) {
 			},
 			wantErr: false,
 		},
+		{
+			name: "Extra hosts",
+			args: args{requirements: []sdk.Requirement{{Name: "go-official-1.9.1", Type: sdk.ModelRequirement, Value: "golang:1.9.1 --port=8080:8081/tcp --privileged --port=9080:9081/tcp --add-host=aaa:1.2.3.4 --add-host=bbb:5.6.7.8"}}},
+			want: &dockerOpts{
+				privileged: true,
+				ports: nat.PortMap{
+					nat.Port("8081/tcp"): []nat.PortBinding{{HostIP: "0.0.0.0", HostPort: "8080"}},
+					nat.Port("9081/tcp"): []nat.PortBinding{{HostIP: "0.0.0.0", HostPort: "9080"}},
+				},
+				extraHosts: []string{"aaa:1.2.3.4", "bbb:5.6.7.8"},
+			},
+			wantErr: false,
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {

engine/hatchery/swarm/swarm_util_kill.go

@@ -79,14 +79,14 @@ func (h *HatcherySwarm) killAwolNetworks() error {
 	//Checking networks
 	nets, errLN := h.dockerClient.NetworkList(context.Background(), types.NetworkListOptions{})
 	if errLN != nil {
-		log.Warning("killAwolWorker> Cannot get networks: %s", errLN)
+		log.Warning("killAwolNetworks> Cannot get networks: %s", errLN)
 		return errLN
 	}
 
 	for i := range nets {
 		n, err := h.dockerClient.NetworkInspect(context.Background(), nets[i].ID)
 		if err != nil {
-			log.Warning("killAwolWorker> Unable to get network info: %v", err)
+			log.Warning("killAwolNetworks> Unable to get network info: %v", err)
 			continue
 		}
 
@@ -102,9 +102,9 @@ func (h *HatcherySwarm) killAwolNetworks() error {
 			continue
 		}
 
-		log.Debug("killAwolWorker> Delete network %s", n.Name)
+		log.Debug("killAwolNetworks> Delete network %s", n.Name)
 		if err := h.dockerClient.NetworkRemove(context.Background(), n.ID); err != nil {
-			log.Warning("killAwolWorker> Unable to delete network %s err:%s", n.Name, err)
+			log.Warning("killAwolNetworks> Unable to delete network %s err:%s", n.Name, err)
 		}
 	}
 	return nil

engine/hatchery/swarm/swarm_util_pull.go

@@ -25,8 +25,5 @@ func (h *HatcherySwarm) pullImage(img string, timeout time.Duration) error {
 
 	btes, _ := ioutil.ReadAll(res)
 	log.Debug("pullImage> %s", string(btes))
-	if err := res.Close(); err != nil {
-		return err
-	}
-	return nil
+	return res.Close()
 }

engine/worker/requirement.go

@@ -186,7 +186,7 @@ func checkMemoryRequirement(w *currentWorker, r sdk.Requirement) (bool, error) {
 
 func checkVolumeRequirement(w *currentWorker, r sdk.Requirement) (bool, error) {
 	// available only on worker booked
-	if w.bookedPBJobID == 0 || w.bookedWJobID == 0 {
+	if w.bookedPBJobID == 0 && w.bookedWJobID == 0 {
 		return false, nil
 	}
 

ui/src/app/views/settings/worker-model/add/worker-model.add.html

@@ -81,8 +81,8 @@ <h4 class="ui header"><i class="fa fa-book fa-3x"></i></h4>
 
             <i class="fa fa-question-circle"></i> {{'worker_model_help_howtos' | translate}}
             <div class="ui list">
-              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker-simple/">{{'worker_model_help_howto_link_1' | translate}}</a>
-              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker-customized/">{{'worker_model_help_howto_link_2' | translate}}</a>
+              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/">{{'worker_model_help_howto_link_1' | translate}}</a>
+              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/docker-customized/">{{'worker_model_help_howto_link_2' | translate}}</a>
               <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/openstack/">{{'worker_model_help_howto_link_3' | translate}}</a>
               <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/">{{'worker_model_help_howto_link_4' | translate}}</a>
             </div>

ui/src/app/views/settings/worker-model/edit/worker-model.edit.html

@@ -119,10 +119,10 @@ <h4 class="ui header"><i class="fa fa-book fa-3x"></i></h4>
 
             <i class="fa fa-question-circle"></i> {{'worker_model_help_howtos' | translate}}
             <div class="ui list">
-              <a class="item" href="https://ovh.github.io/cds/tutorials/tutorials.worker-model-docker-simple/">{{'worker_model_help_howto_link_1' | translate}}</a>
-              <a class="item" href="https://ovh.github.io/cds/tutorials/tutorials.worker-model-docker-customized/">{{'worker_model_help_howto_link_2' | translate}}</a>
-              <a class="item" href="https://ovh.github.io/cds/tutorials/tutorials.worker-model-openstack/">{{'worker_model_help_howto_link_3' | translate}}</a>
-              <a class="item" href="https://ovh.github.io/cds/advanced/advanced.worker.model/">{{'worker_model_help_howto_link_4' | translate}}</a>
+              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/">{{'worker_model_help_howto_link_1' | translate}}</a>
+              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/docker-customized/">{{'worker_model_help_howto_link_2' | translate}}</a>
+              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/openstack/">{{'worker_model_help_howto_link_3' | translate}}</a>
+              <a class="item" target="_blank" href="https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/">{{'worker_model_help_howto_link_4' | translate}}</a>
             </div>
           </div>
         </div>

ui/src/assets/i18n/en.json

@@ -494,7 +494,7 @@
   "requirement_type": "Type",
   "requirement_value": "Value",
   "requirement_help_binary": "Requirement type 'binary': CDS will choose a worker with this binary in his path.",
-  "requirement_help_model": "Requirement type 'model': <ul><li>If you select a <a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Worker Model</a>, CDS will launch your job inside it</li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker-simple/\">Create a worker model based on a docker image from Docker Hub</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker-customized/\">Create a worker model with your own image</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/openstack/\">Create a worker model based on a Openstack image</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Read more</a></li></ul>",
+  "requirement_help_model": "Requirement type 'model': <ul><li>If you select a <a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Worker Model</a>, CDS will launch your job inside it</li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/\">Create a worker model based on a docker image from Docker Hub</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/docker-customized/\">Create a worker model with your own image</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/openstack/\">Create a worker model based on a Openstack image</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Read more</a></li></ul>",
   "requirement_help_memory": "Requirement type 'memory': <ul><li>If you want 4Go, enter value in Mo: <b>4096</b></li><li>Memory requirement is availabe only on <a href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Worker Model</a> type Docker</li></ul>",
   "requirement_help_network": "Requirement type 'network': <ul><li>CDS will choose a worker which can reach this IP.</li></ul>",
   "requirement_help_hostname": "Requirement type 'hostname': <ul><li>This Job will be take by a worker hosted on this host</li></ul>",

ui/src/assets/i18n/fr.json

@@ -494,7 +494,7 @@
   "requirement_type": "Type",
   "requirement_value": "Valeur",
   "requirement_help_binary": "Pré-requis type 'binary': CDS choisira un worker possédant ce binaire dans son PATH.",
-  "requirement_help_model": "Pré-requis type 'model': <ul><li>Si vous sélectionnez un <a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Worker Model</a>, CDS lancera votre Job dans une instance de celui-ci</li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker-simple/\">Créer un modèle de worker en utilisant une image depuis Docker Hub</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker-customized/\">Créer un modèle de worker avec votre propre image docker</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/openstack/\">Créer un modèle de worker Openstack</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">En savoir plus</a></li></ul>",
+  "requirement_help_model": "Pré-requis type 'model': <ul><li>Si vous sélectionnez un <a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Worker Model</a>, CDS lancera votre Job dans une instance de celui-ci</li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/\">Créer un modèle de worker en utilisant une image depuis Docker Hub</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/docker/docker-customized/\">Créer un modèle de worker avec votre propre image docker</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/openstack/\">Créer un modèle de worker Openstack</a></li><li><a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">En savoir plus</a></li></ul>",
   "requirement_help_memory": "Pré-requis type 'memory': <ul><li>Si vous souhaitez 5Go, entrez la valeur suivante: <b>4096</b></li><li>Le prérequis memory est disponible uniquement avec les <a target=\"_blank\" href=\"https://ovh.github.io/cds/workflows/pipelines/requirements/worker-model/\">Worker Model</a> de type Docker</li></ul>",
   "requirement_help_network": "Pré-requis type 'network': <ul><li>CDS choisira un worker qui pourra atteindre cette IP</li></ul>",
   "requirement_help_hostname": "Pré-requis type 'hostname': <ul><li>Ce job sera lancé par un worker possédant ce Hostname</li></ul>",