Overrides for run now requests

[pschoenfelder]

No description provided.

[pschoenfelder]

Full disclosure - I have no idea if this touches all the things it needs to. I added a unit test to check that environment variable overrides specified in a run-now request are propagated through to a Mesos task, but I don't know if this covers all the workflows (I doubt it does). Do you know if/what other things need to pick up on the overrides?

[pschoenfelder]

Also, I added a builder for run-now requests for the unit tests since there was a lot of Optional.absent() boilerplate going on, but I only added it to the test package. Would it be useful elsewhere?

[ssalinas]

Would be great to add this to SingularityBase 👍 . For context, we've debated moving to immutables to get more of our builders/etc generated automatically, but haven't made the move yet. For any classes whose constructors start to get larger, it's worth making a builder

[pschoenfelder]

Awesome! Will move it

[ssalinas]

maybe environmentOverrides instead? I think we would want these to take precedence over the data in the SingularityDeploy in the case where there are two keys with different values.

[pschoenfelder]

Yup - definitely more clear

[tpetr]

SingularityTask uses env for the env var map -- should we follow that convention?

[ssalinas]

Good point, SingularityDeploy does as well.

[ssalinas]

Rather than having an optional map, it makes more sense to just default it to empty. Maps have a nice default state of empty that we can use to keep things cleaner and cut down on all of the isPresent checks everywhere. So, in the constructor it'd be something like this. environmentOverrides = environmentOverrides == null ? Collections.emptyMap() : environmentOverrides

[ssalinas]

Why change this to Object from String?

[pschoenfelder]

Think it was more useful as I was developing. Looking at it now though, no good reason. I'll revert it.

[ssalinas]

It seems like duplicate work to do this at the end. The helpful piece of the builder is that we can put it together along the way and not have to iterate through the map again, which is why there are all of the setEnv calls above.

[pschoenfelder]

My unit test was catching an error scenario where an overridden variable was duplicated in the Environment.Builder. I didn't find anything in the docs to indicate that there's anything preventing this or a clean way to handle dupes, so I stuck everything in a map first. Do you know if there's a better way to do this without the map?

[ssalinas]

generally try to keep HubSpot-specific things out of Singularity, even for tests ;)

[pschoenfelder]

👍 Good catch - didn't think of that

[ssalinas]

could probably just do the for loop here. for (Variable var : task.getMesosTask().getCommand().getEnvironment().getVariablesList()) and just do the assert equals against getName/getKey. Saves you streaming to a map, only to then iterate.

[pschoenfelder]

True, but the mesos task variables are a superset of the override variables, so the loop would require some kind of checking like this:

    Boolean checkedStartedByUser = false;

    for (Variable var : task.getMesosTask().getCommand().getEnvironment().getVariablesList()) {
      if (overrideVariables.containsKey(var.getName())) {
        if (var.getName().equals("MY_NEW_ENV_VAR")) {
          checkedMyNewEnvVar = true;
        } else if (var.getName().equals("STARTED_BY_USER")) {
          checkedStartedByUser = true;
        }

        assertEquals(
            "Environment variable " + var + " not overridden.",
            var.getValue(),
            overrideVariables.get(var.getName()));
      }
    }
    assertTrue(checkedMyNewEnvVar);
    assertTrue(checkedStartedByUser);

Which do you think is cleaner (or is there an even cleaner alternative?)?

[ssalinas]

This particular field specifies to run immediately with these settings upon successful completion of a deploy. In this case it isn't actually getting transferred to your pending task

[ssalinas]

As mentioned below, this is a field that specifies config to run immediately upon successful deploy. The SingularityPendingTask is where the environment override needs to be trasferred to first. So, something like:

• New RunNowRequest comes in with environmentOverrides and the SingularityRunNowRequest is saved as part of the SingularityPendingRequest and put into the pending queue (triggered from the RequestResource
• overrides from RunNowRequest saved as part of the SingularityPendingTask when it is created based on that pending request from the queue (this is the piece that is currently missing)
• When the task builder is ready to build the task from the pending task, it can then read the environment overrides from the pending task and apply those here

[pschoenfelder]

Next diff will have changes that I believe satisfy this, though I'm not sure my implementation is as clean as it could be. Either way, let me know and I'll work through it.

[ssalinas]

The pendingTask being used here is the one created in the Before. Since the environment overrides will be on the pending task (see more detail in task builder comments), we'll need to construct our own pending task to test this here

[ssalinas]

Good to add a toString on these. Ends up being helpful for debugging later if we ever have to log it out.

[pschoenfelder]

Good to know :)

[ssalinas]

same toString comment for this one

[ssalinas]

the @JsonProperty on these needs to stay as it was before. This is one of the pieces jackson uses to determine the field name in the json. e.g. @JsonProperty("message") becomes {"message": {}}

[pschoenfelder]

Hmmm...think that was my improper use of Intellij's refactoring tools. Going to need to be more careful of that in the future.

[ssalinas]

similar comment here, don't want to rename these to set...

[ssalinas]

empty map instead of null here?

[ssalinas]

this should be the last missing piece of the puzzle to connect it all the way through. The SingularityPendingRequest is the object that will carry the envOverrides from the RunNowRequest to the pending task.

In RequestResource.scheduleImmediately we create a SingularityPendingRequest from the provided SingularityRunNowRequest. That pending request will need to take in the environment overrides, so that we can later use them on this line to put them on the pending task.

The RequestResource calls checkRunNowRequest on the SingularityValidator, which is what actually constructs the object actually running some checks. That is where the envOverrides will need to be put on the SingularityPendingRequest

[ssalinas]

From what I can see this looks good. Thanks for those builder simplifications as well 🎉

I think we should be good to give this one a go in hs_staging if it was working for you locally

[ssalinas]

indent ended up a bit funny here

[baconmania]

Quick note here, we don't want to modify existing constructor signatures for classes in a Base module of a project (or more generally, for any classes that are intended for use outside the project itself), because it'd break other projects. Adding "proxy" constructors is one way around this.

[baconmania]

Should we be handling the case where pendingType == null here?

[pschoenfelder]

I don't think so. This is just a small refactor of the previous code, which would do nothing in the case where the new code has pendingType == null.

[ssalinas]

🚢

[ssalinas]

🚢

[pschoenfelder]

🚢