Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/docker params #538

Merged
merged 3 commits into from
Sep 11, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ Here is a more elaborate example of a dependent job hash:
## Adding a Docker Job

A docker job takes the same format as a scheduled job or a dependency job and runs on a Docker container.
To configure it, an additional `container` argument is required, which contains a type (required), an image (required), a network mode (optional), mounted volumes (optional) and whether Mesos should always pull the latest image before executing or not (optional).
To configure it, an additional `container` argument is required, which contains a type (required), an image (required), a network mode (optional), mounted volumes (optional), parameters (optional) and whether Mesos should always pull the latest image before executing or not (optional).

* Endpoint: __/scheduler/iso8601__ or __/scheduler/dependency__
* Method: __POST__
Expand Down Expand Up @@ -229,6 +229,20 @@ Chronos by adding the `forcePullImage` boolean to your `container` configuration
Chronos will default to not doing a `docker pull` if the image is already found on the executing node. The alternative approach is to use versions/tags for
your images.

There is also support for passing in arbitrary docker config options.

```json
{
"container": {
"type": "DOCKER",
"image": "libmesos/ubuntu",
"parameters": [
{ "key": "a-docker-option", "value": "xxx" },
{ "key": "b-docker-option", "value": "yyy" }
]
}
```

## Updating Task Progress

Task progress can be updated by providing the number of additional elements processed. This will increment the existing count of elements processed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,5 +27,6 @@ case class Volume(
case class DockerContainer(
@JsonProperty image: String,
@JsonProperty volumes: Seq[Volume],
@JsonProperty parameters: Seq[Parameter],
@JsonProperty network: NetworkMode = NetworkMode.HOST,
@JsonProperty forcePullImage: Boolean = false)
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package org.apache.mesos.chronos.scheduler.jobs

import com.fasterxml.jackson.annotation.JsonProperty

import org.apache.mesos.{ Protos => mesos }

/**
* Represents an environment variable definition for the job
*/
case class Parameter(
key: String,
value: String) {

def toProto(): mesos.Parameter =
mesos.Parameter.newBuilder
.setKey(key)
.setValue(value)
.build
}
object Parameter {
def apply(proto: mesos.Parameter): Parameter =
Parameter(
proto.getKey,
proto.getValue
)
}
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ class MesosTaskBuilder @Inject()(val conf: SchedulerConfiguration) {
.setImage(job.container.image)
.setNetwork(DockerInfo.Network.valueOf(job.container.network.toString.toUpperCase))
.setForcePullImage(job.container.forcePullImage)
.addAllParameters(job.container.parameters.map(_.toProto).asJava)
.build()).build
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,16 @@ class JobDeserializer extends JsonDeserializer[BaseJob] {
if (containerNode.has("forcePullImage") && containerNode.get("forcePullImage") != null)
Try(containerNode.get("forcePullImage").asText.toBoolean).getOrElse(false)
else false
container = DockerContainer(containerNode.get("image").asText, volumes, networkMode, forcePullImage)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit: can you drop the extra newline?

var parameters = scala.collection.mutable.ListBuffer[Parameter]()
if (containerNode.has("parameters")) {
containerNode.get("parameters").elements().map {
case node: ObjectNode =>
Parameter(node.get("key").asText(), node.get("value").asText)
}.foreach(parameters.add)
}

container = DockerContainer(containerNode.get("image").asText, volumes, parameters, networkMode, forcePullImage)
}

val constraints = scala.collection.mutable.ListBuffer[Constraint]()
Expand Down
13 changes: 13 additions & 0 deletions src/main/scala/org/apache/mesos/chronos/utils/JobSerializer.scala
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,19 @@ class JobSerializer extends JsonSerializer[BaseJob] {
json.writeEndArray()
json.writeFieldName("forcePullImage")
json.writeBoolean(baseJob.container.forcePullImage)

json.writeFieldName("parameters")
json.writeStartArray()
baseJob.container.parameters.foreach { v =>
json.writeStartObject()
json.writeFieldName("key")
json.writeString(v.key)
json.writeFieldName("value")
json.writeString(v.value)
json.writeEndObject()
}
json.writeEndArray()

json.writeEndObject()
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,10 @@ class SerDeTest extends SpecificationWithJUnit {
)

val forcePullImage = false
val container = DockerContainer("dockerImage", volumes, NetworkMode.BRIDGE, forcePullImage)

var parameters = scala.collection.mutable.ListBuffer[Parameter]()

val container = DockerContainer("dockerImage", volumes, parameters, NetworkMode.BRIDGE, forcePullImage)

val arguments = Seq(
"-testOne"
Expand Down Expand Up @@ -69,7 +72,9 @@ class SerDeTest extends SpecificationWithJUnit {
)

val forcePullImage = true
val container = DockerContainer("dockerImage", volumes, NetworkMode.HOST, forcePullImage)
var parameters = scala.collection.mutable.ListBuffer[Parameter]()

val container = DockerContainer("dockerImage", volumes, parameters, NetworkMode.HOST, forcePullImage)

val arguments = Seq(
"-testOne"
Expand Down