Overrides for run now requests by pschoenfelder · Pull Request #1650 · HubSpot/Singularity

pschoenfelder · 2017-11-14T16:55:27Z

No description provided.

…t or a custom executor.

Also some test refactoring

pschoenfelder · 2017-11-14T16:55:40Z

Full disclosure - I have no idea if this touches all the things it needs to. I added a unit test to check that environment variable overrides specified in a run-now request are propagated through to a Mesos task, but I don't know if this covers all the workflows (I doubt it does). Do you know if/what other things need to pick up on the overrides?

pschoenfelder · 2017-11-14T16:58:36Z

Also, I added a builder for run-now requests for the unit tests since there was a lot of Optional.absent() boilerplate going on, but I only added it to the test package. Would it be useful elsewhere?

ssalinas · 2017-11-14T17:36:33Z

+import com.hubspot.mesos.Resources;
+import com.hubspot.singularity.api.SingularityRunNowRequest;
+
+public class SingularityRunNowRequestBuilder {


Would be great to add this to SingularityBase 👍 . For context, we've debated moving to immutables to get more of our builders/etc generated automatically, but haven't made the move yet. For any classes whose constructors start to get larger, it's worth making a builder

Awesome! Will move it

ssalinas · 2017-11-14T17:37:18Z

  private final Optional<List<String>> commandLineArgs;
  private final Optional<Boolean> skipHealthchecks;
  private final Optional<Resources> resources;
+  private final Optional<Map<String, String>> environmentVariables;


maybe environmentOverrides instead? I think we would want these to take precedence over the data in the SingularityDeploy in the case where there are two keys with different values.

Yup - definitely more clear

SingularityTask uses env for the env var map -- should we follow that convention?

Good point, SingularityDeploy does as well.

ssalinas · 2017-11-14T17:39:16Z

                                  @JsonProperty("runId") Optional<String> runId,
                                  @JsonProperty("commandLineArgs") Optional<List<String>> commandLineArgs,
                                  @JsonProperty("resources") Optional<Resources> resources,
+                                  @JsonProperty("environmentVariables") Optional<Map<String, String>> environmentVariables,


Rather than having an optional map, it makes more sense to just default it to empty. Maps have a nice default state of empty that we can use to keep things cleaner and cut down on all of the isPresent checks everywhere. So, in the constructor it'd be something like this. environmentOverrides = environmentOverrides == null ? Collections.emptyMap() : environmentOverrides

ssalinas · 2017-11-14T17:45:17Z

  }

-  private void setEnv(Environment.Builder envBldr, String key, Object value) {
+  private void setEnv(Environment.Builder envBldr, Object key, Object value) {


Why change this to Object from String?

Think it was more useful as I was developing. Looking at it now though, no good reason. I'll revert it.

ssalinas · 2017-11-14T17:49:19Z

+      }
    }

+    Environment.Builder envBldr = Environment.newBuilder();


It seems like duplicate work to do this at the end. The helpful piece of the builder is that we can put it together along the way and not have to iterate through the map again, which is why there are all of the setEnv calls above.

My unit test was catching an error scenario where an overridden variable was duplicated in the Environment.Builder. I didn't find anything in the docs to indicate that there's anything preventing this or a clean way to handle dupes, so I stuck everything in a map first. Do you know if there's a better way to do this without the map?

ssalinas · 2017-11-14T18:04:13Z


+  @Test
+  public void testEnvironmentVariableOverrides() {
+    Map<String, String> overrideVariables = new HashMap<>();


generally try to keep HubSpot-specific things out of Singularity, even for tests ;)

👍 Good catch - didn't think of that

ssalinas · 2017-11-14T18:06:08Z

+        .build();
+    final SingularityTaskRequest taskRequest = new SingularityTaskRequest(request, deploy, pendingTask);
+    final SingularityTask task = builder.buildTask(offerHolder, null, taskRequest, taskResources, executorResources);
+


could probably just do the for loop here. for (Variable var : task.getMesosTask().getCommand().getEnvironment().getVariablesList()) and just do the assert equals against getName/getKey. Saves you streaming to a map, only to then iterate.

True, but the mesos task variables are a superset of the override variables, so the loop would require some kind of checking like this:

Boolean checkedStartedByUser = false; for (Variable var : task.getMesosTask().getCommand().getEnvironment().getVariablesList()) { if (overrideVariables.containsKey(var.getName())) { if (var.getName().equals("MY_NEW_ENV_VAR")) { checkedMyNewEnvVar = true; } else if (var.getName().equals("STARTED_BY_USER")) { checkedStartedByUser = true; } assertEquals( "Environment variable " + var + " not overridden.", var.getValue(), overrideVariables.get(var.getName())); } } assertTrue(checkedMyNewEnvVar); assertTrue(checkedStartedByUser);

Which do you think is cleaner (or is there an even cleaner alternative?)?

ssalinas · 2017-11-14T18:07:36Z

+    final SingularityRequest request = new SingularityRequestBuilder("test", RequestType.WORKER)
+        .build();
+    final SingularityDeploy deploy = new SingularityDeployBuilder("test", "1")
+        .setCommand(Optional.of("/bin/echo hi"))


This particular field specifies to run immediately with these settings upon successful completion of a deploy. In this case it isn't actually getting transferred to your pending task

ssalinas · 2017-11-14T18:11:59Z

+    }

+    if (task.getDeploy().getRunImmediately().isPresent()) {
+      SingularityRunNowRequest runNowRequest = task.getDeploy().getRunImmediately().get();


As mentioned below, this is a field that specifies config to run immediately upon successful deploy. The SingularityPendingTask is where the environment override needs to be trasferred to first. So, something like:

New RunNowRequest comes in with environmentOverrides and the SingularityRunNowRequest is saved as part of the SingularityPendingRequest and put into the pending queue (triggered from the RequestResource

overrides from RunNowRequest saved as part of the SingularityPendingTask when it is created based on that pending request from the queue (this is the piece that is currently missing)

When the task builder is ready to build the task from the pending task, it can then read the environment overrides from the pending task and apply those here

Next diff will have changes that I believe satisfy this, though I'm not sure my implementation is as clean as it could be. Either way, let me know and I'll work through it.

ssalinas · 2017-11-14T18:12:46Z

+    final SingularityDeploy deploy = new SingularityDeployBuilder("test", "1")
+        .setCommand(Optional.of("/bin/echo hi"))
+        .setRunImmediately(Optional.of(runNowRequest))
+        .build();


The pendingTask being used here is the one created in the Before. Since the environment overrides will be on the pending task (see more detail in task builder comments), we'll need to construct our own pending task to test this here

ssalinas · 2017-11-15T19:22:31Z

+        pendingTaskId, cmdLineArgsList, user, runId, skipHealthchecks, message, resources, envOverrides, actionId
+    );
+  }
+


Good to add a toString on these. Ends up being helpful for debugging later if we ever have to log it out.

Good to know :)

ssalinas · 2017-11-15T19:22:43Z

+    return new SingularityRunNowRequest(
+        message, skipHealthchecks, runId, commandLineArgs, resources, envOverrides, runAt);
+  }
+


same toString comment for this one

ssalinas · 2017-11-15T19:24:21Z

-                                  @JsonProperty("skipHealthchecks") Optional<Boolean> skipHealthchecks,
-                                  @JsonProperty("runId") Optional<String> runId,
-                                  @JsonProperty("commandLineArgs") Optional<List<String>> commandLineArgs,
+  public SingularityRunNowRequest(@JsonProperty("setMessage") Optional<String> message,


the @JsonProperty on these needs to stay as it was before. This is one of the pieces jackson uses to determine the field name in the json. e.g. @JsonProperty("message") becomes {"message": {}}

Hmmm...think that was my improper use of Intellij's refactoring tools. Going to need to be more careful of that in the future.

ssalinas · 2017-11-15T19:24:42Z

  }

-  @ApiModelProperty(required=false, value="Override the resources from the active deploy for this run")
+  @ApiModelProperty(required=false, value="Override the setResources from the active deploy for this run")


similar comment here, don't want to rename these to set...

ssalinas · 2017-11-15T19:25:25Z

+    this.commandLineArgs = Optional.absent();
+    this.skipHealthchecks = Optional.absent();
+    this.resources = Optional.absent();
+    this.envOverrides = null;


empty map instead of null here?

ssalinas · 2017-11-15T19:33:30Z

+              pendingRequest.getSkipHealthchecks(),
+              pendingRequest.getMessage(),
+              pendingRequest.getResources(),
+              null,


this should be the last missing piece of the puzzle to connect it all the way through. The SingularityPendingRequest is the object that will carry the envOverrides from the RunNowRequest to the pending task.

In RequestResource.scheduleImmediately we create a SingularityPendingRequest from the provided SingularityRunNowRequest. That pending request will need to take in the environment overrides, so that we can later use them on this line to put them on the pending task.

The RequestResource calls checkRunNowRequest on the SingularityValidator, which is what actually constructs the object actually running some checks. That is where the envOverrides will need to be put on the SingularityPendingRequest

ssalinas · 2017-11-16T16:02:18Z

From what I can see this looks good. Thanks for those builder simplifications as well 🎉

I think we should be good to give this one a go in hs_staging if it was working for you locally

ssalinas · 2017-11-21T19:01:11Z

+              new SingularityPendingTaskId(
+                  request.getId(), deployId, nextRunAt.get(), nextInstanceNumber,
+                  pendingRequest.getPendingType(), pendingRequest.getTimestamp()),
+              pendingRequest.getCmdLineArgsList(),


indent ended up a bit funny here

baconmania · 2017-11-21T19:43:12Z

      Optional<List<String>> commandLineArgs,
-      Optional<Resources> resources
+      Optional<Resources> resources,
+      Map<String, String> envOverrides


Quick note here, we don't want to modify existing constructor signatures for classes in a Base module of a project (or more generally, for any classes that are intended for use outside the project itself), because it'd break other projects. Adding "proxy" constructors is one way around this.

baconmania · 2017-11-21T20:04:10Z

+          pendingType = PendingType.NEW_DEPLOY;
        }
      }
+      if (pendingType != null) {


Should we be handling the case where pendingType == null here?

I don't think so. This is just a small refactor of the previous code, which would do nothing in the case where the new code has pendingType == null.

Support user overrides in Run-Now requests.

ssalinas · 2017-12-05T20:55:13Z

🚢

ssalinas · 2017-12-06T19:21:07Z

🚢

pschoenfelder · 2017-12-07T16:02:38Z

🚢

baconmania and others added 11 commits November 3, 2017 15:38

Support extra artifacts in models.

4d1815c

Download additional artifacts as well.

edab7aa

Update tests.

27ddc68

Default extraArtifacts to an empty list instead of using Optional<>.

3634f26

Explicit type not required here.

4bf55d2

Don't use an immutable list here.

cf38987

Safest way of doing this.

7db038c

Download extra artifacts regardless of whether we're using the defaul…

f9d59e5

…t or a custom executor.

Allow clients to set an output file path.

42fe3b6

Failing test case for environment variable overrides in run now requests

6c0c836

Also some test refactoring

Passing of environment variable overrides through mesos task builder

4b1c9fa

pschoenfelder requested review from baconmania and ssalinas November 14, 2017 16:55

ssalinas suggested changes Nov 14, 2017

View reviewed changes

CR changes

d81a6a9

ssalinas added this to the 0.19.0 milestone Nov 15, 2017

Added environment variable overrides into pending tasks

f3012b7

ssalinas suggested changes Nov 15, 2017

View reviewed changes

Paul Schoenfelder added 3 commits November 15, 2017 15:20

Added environment variable overrides into pending tasks

a049250

CR Changes

5a2e951

Added environment variable overrides to pending requests

19267fb

ssalinas suggested changes Nov 21, 2017

View reviewed changes

ssalinas added the hs_staging label Nov 21, 2017

Indentation fix

03b9972

baconmania reviewed Nov 21, 2017

View reviewed changes

Merge branch 'run-now-with-additional-artifacts' into run_now_overrides

3110897

baconmania and others added 2 commits November 21, 2017 15:20

Don't break compatibility.

38554c0

Log warning when pending type cannot be determine

547b834

pschoenfelder added the hs_qa label Nov 28, 2017

ssalinas mentioned this pull request Dec 1, 2017

Run-now with additional artifacts. #1644

Closed

ssalinas changed the title ~~Environment variable overrides for run-now requests~~ Overrides for run now requests Dec 1, 2017

baconmania and others added 4 commits December 1, 2017 17:48

Support user overrides in Run-Now requests.

19b385e

Unnecessary method.

30a441e

Fix isPresent() check.

28e9b53

Merge pull request #1668 from HubSpot/run-now-user-override

097469d

Support user overrides in Run-Now requests.

baconmania added 2 commits December 6, 2017 13:44

Merge branch 'master' into run_now_overrides

bc231b2

Fix test.

15c44da

ssalinas added the hs_stable label Dec 6, 2017

fix merge conflicts in task builder

265f042

ssalinas merged commit 54616e5 into master Dec 7, 2017

ssalinas deleted the run_now_overrides branch December 7, 2017 16:24

Conversation

pschoenfelder commented Nov 14, 2017

Uh oh!

pschoenfelder commented Nov 14, 2017

Uh oh!

pschoenfelder commented Nov 14, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pschoenfelder Nov 14, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ssalinas commented Nov 16, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ssalinas commented Dec 5, 2017

Uh oh!

ssalinas commented Dec 6, 2017

Uh oh!

pschoenfelder commented Dec 7, 2017

Uh oh!

Reviewers

pschoenfelder Nov 14, 2017 •

edited

Loading