Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-8490] [mesos] Allow custom docker parameters for docker tasks on Mesos #5346

Closed
wants to merge 1 commit into from
Closed

[FLINK-8490] [mesos] Allow custom docker parameters for docker tasks on Mesos #5346

wants to merge 1 commit into from

Conversation

lishim
Copy link
Contributor

@lishim lishim commented Jan 23, 2018

What is the purpose of the change

  • This pull request introduces a feature that allows passing custom parameters to docker when using docker containerizer.

Brief change log

  • A new parameter mesos.resourcemanager.tasks.container.docker.parameters is introduced

Verifying this change

This change added tests and can be verified as follows:

  • Added unit tests that verify correct behavior of the new parameter
  • Manually verified the change by running it in an Mesos cluster with Docker containerizer and 2 task managers, setting mesos.resourcemanager.tasks.container.docker.parameters=privileged=true and verifying that the task manager container is able to run a privileged command, then removing the parameter and verifying that the privileged command can no longer run.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: yes
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs

@joerg84
Copy link
Contributor

joerg84 commented Jan 23, 2018

Looks great! I need to test that tomorrow in more detail.
@tillrohrmann @EronWright

@EronWright
Copy link
Contributor

+1

joerg84
joerg84 approved these changes Jan 24, 2018
View reviewed changes
Copy link
Contributor

@joerg84 joerg84 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor comments, should be easy to fix. Overall impressive work!

docs/ops/config.md Outdated
@@ -508,6 +508,8 @@ May be set to -1 to disable this feature.

- `mesos.resourcemanager.tasks.container.volumes`: A comma separated list of `[host_path:]`container_path`[:RO|RW]`. This allows for mounting additional volumes into your container. (**NO DEFAULT**)

- `mesos.resourcemanager.tasks.container.docker.parameters`: Custom parameters to be passed into docker run command when using the docker containerizer. Comma separated list of key=value pairs. (**NO DEFAULT**)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 blanks between "containerizer. Comma"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

docs/ops/deployment/mesos.md Outdated
@@ -264,6 +264,8 @@ May be set to -1 to disable this feature.

`mesos.resourcemanager.tasks.container.volumes`: A comma separated list of `[host_path:]`container_path`[:RO|RW]`. This allows for mounting additional volumes into your container. (**NO DEFAULT**)

`mesos.resourcemanager.tasks.container.docker.parameters`: Custom parameters to be passed into docker run command when using the docker containerizer. Comma separated list of key=value pairs. (**NO DEFAULT**)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

...est/java/org/apache/flink/mesos/runtime/clusterframework/MesosTaskManagerParametersTest.java Outdated
config.setString(MesosTaskManagerParameters.MESOS_RM_CONTAINER_DOCKER_PARAMETERS, "testKey=testValue");

MesosTaskManagerParameters params = MesosTaskManagerParameters.create(config);
assertEquals(params.dockerParameters().get(0).getKey(), "testKey");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe check the size first, same below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a size check.

...rc/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosTaskManagerParameters.java Outdated

for (String dockerParameterSpecification : dockerParameterSpecifications) {
if (!dockerParameterSpecification.trim().isEmpty()) {
String[] match = dockerParameterSpecification.split("=", 2);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use split without the limit, i.e., .split("=")

  1. You already test for length below
  2. this way you would miss invalid specifications such as key2=val1=val3

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some Docker parameters are key=value pairs themselves, so key2=val1=val3 is actually valid specification. I added a comment to clarify this, and included a test case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for the explanation!

@lishim
Copy link
Contributor Author

lishim commented Jan 24, 2018

Thanks @joerg84 for your review and comments. I have updated the PR to address your concerns.

@joerg84
Copy link
Contributor

joerg84 commented Jan 25, 2018

@tillrohrmann Could you take a final look?

tillrohrmann
tillrohrmann approved these changes Jan 26, 2018
View reviewed changes
Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution @lishim and the reviews @joerg84 and @EronWright. Changes look good to me. Merging this PR.

@asfgit asfgit closed this in 6969fe2 Jan 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joerg84 joerg84 joerg84 approved these changes

@tillrohrmann tillrohrmann tillrohrmann approved these changes

Assignees
No one assigned
Labels
component=Deployment/Mesos
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@lishim @joerg84 @EronWright @tillrohrmann @rmetzger