Deleting state #1386

matush-v · 2016-12-27T19:37:39Z

Creates a new request state DELETING to better know when a request is still going through cleanup. Request State begins as DELETING and is changed to DELETED once all tasks for the request are cleaned. Currently using USER_REQUESTED_DESTROY to determine if the task is a delete request.

@tpetr @ssalinas

ssalinas · 2017-01-03T16:19:52Z

SingularityService/src/main/java/com/hubspot/singularity/data/RequestManager.java

+    return save(request, RequestState.DELETING, historyType, timestamp, user, message);
+  }
+
+  public SingularityCreateResult deleted(SingularityRequest request, RequestHistoryType historyType, long timestamp, Optional<String> user, Optional<String> message) {


maybe for these methods we name them more like an action. i.e. markDeleted or markDeleting, just to clarify what they are doing. Especially since we already have a deleteRequest method (which could maybe be renamed to something like startRequestDelete?)

ssalinas · 2017-01-03T16:22:01Z

SingularityService/src/main/java/com/hubspot/singularity/scheduler/SingularityCleaner.java

-            Optional<SingularityRequestHistory> maybeHistory = requestHistoryHelper.getLastHistory(requestId);
-            if (maybeHistory.isPresent() && maybeHistory.get().getRequest().isLoadBalanced() && configuration.isDeleteRemovedRequestsFromLoadBalancer()) {
-              createLbCleanupRequest(requestId, matchingActiveTaskIds);
+            if (matchingActiveTaskIds.iterator().hasNext()) {


Maybe Iterables.isEmpty(matchingActiveTaskIds) just for cleanliness/readability

ssalinas · 2017-01-03T16:42:41Z

SingularityService/src/main/java/com/hubspot/singularity/scheduler/SingularityCleaner.java

+
+  private void cleanupRequestIfNoRemainingTasks(SingularityTaskId taskId, Map<String, List<String>> requestIdToTaskIds) {
+    List<String> requestTasks = requestIdToTaskIds.get(taskId.getRequestId());
+    requestTasks.remove(taskId.getId());


requestIdToTaskIds isn't guaranteed to have the key present, will want to check that or this line will cause NPEs

ah, I didn't notice we would hit this method for other cleanup types that weren't in the requestIdToTaskIds map. updated to return early if it's null

ssalinas · 2017-01-03T16:42:55Z

Overall looks good, the few comments I left and a test that needs to be fixed:

SingularityHistoryTest.testMessage - count of history items is wrong now that we are creating another history entry

matush-v · 2017-01-09T20:13:01Z

SingularityService/src/main/java/com/hubspot/singularity/data/RequestManager.java

    SingularityDeleteResult deleteResult = delete(getRequestPath(request.getId()));

+    markDeleting(request, RequestHistoryType.DELETED, System.currentTimeMillis(), user, message);
+
    LOG.info("Request {} deleted ({}) by {} - {}", request.getId(), deleteResult, user, message);


@ssalinas should we adjust these log messages too? They're a bit misleading since they kind of jump the gun on the status

Yeah, might be good for the log line to suggest it was 'enqueued for delete' rather than 'deleted'

ssalinas · 2017-01-12T15:27:10Z

SingularityService/src/main/java/com/hubspot/singularity/data/RequestManager.java

@@ -328,9 +334,12 @@ public SingularityDeleteResult deleteRequest(SingularityRequest request, Optiona

    saveHistory(new SingularityRequestHistory(now, user, RequestHistoryType.DELETED, request, message));

+    // moves RequestState to DELETED
    SingularityDeleteResult deleteResult = delete(getRequestPath(request.getId()));


If we want to still be able to grab request data while it is in DELETING state, we shouldn't have this line here. This deletes all of the current request data. We should probably move this step to be triggered by the cleaner after everything is done (i.e. when the active deploy cleanup is happening)

matush-v · 2017-01-12T18:37:52Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularityDeleteResult.java

@@ -1,5 +1,5 @@
 package com.hubspot.singularity;

 public enum SingularityDeleteResult {
-  DELETED, DIDNT_EXIST;
+  DELETING, DELETED, DIDNT_EXIST;


added this state to better reflect the state of the delete when it's first requested

matush-v · 2017-01-12T18:38:19Z

SingularityBase/src/main/java/com/hubspot/singularity/SingularityRequestHistory.java

@@ -16,7 +16,7 @@
  private final Optional<String> message;

  public enum RequestHistoryType {
-    CREATED, UPDATED, DELETED, PAUSED, UNPAUSED, ENTERED_COOLDOWN, EXITED_COOLDOWN, FINISHED, DEPLOYED_TO_UNPAUSE, BOUNCED, SCALED, SCALE_REVERTED;
+    CREATED, UPDATED, DELETING, DELETED, PAUSED, UNPAUSED, ENTERED_COOLDOWN, EXITED_COOLDOWN, FINISHED, DEPLOYED_TO_UNPAUSE, BOUNCED, SCALED, SCALE_REVERTED;


added this state to better reflect the state of the request history

matush-v · 2017-01-12T18:42:27Z

SingularityService/src/main/java/com/hubspot/singularity/scheduler/SingularityCleaner.java

@@ -346,7 +345,7 @@ private void drainRequestCleanupQueue() {
          }
          break;
        case DELETING:
-          if (requestWithState.isPresent()) {
+          if (requestWithState.isPresent() && requestWithState.get().getState() != RequestState.DELETING) {


added this to make sure deleting states aren't ignored and will be cleaned up properly

…deleting-state

ssalinas · 2017-01-13T17:51:10Z

mentioned in person, but to document here as well. Last change needed for this would be to remove the piece of ui code that makes the Remove button redirect you back to the requests page. Should be gtg after that

…deleting-state

matush-v · 2017-01-17T21:05:34Z

@ssalinas cool to move this into staging tmrw morning?

ssalinas · 2017-01-17T21:11:37Z

@darcatron you can just do a then={fetchRequestAndHistory} rather that doing the window.reload in the ui portion. That will trigger a re-fetch of the request data from the api, but we won't need to reload the entire page. After that small change, yeah, think we are good for staging

matush-v · 2017-01-20T23:42:14Z

SingularityService/src/main/java/com/hubspot/singularity/scheduler/SingularityCleaner.java

@@ -458,6 +462,24 @@ private TaskCleanupType pause(SingularityRequestCleanup requestCleanup, Iterable
    return cleanupType;
  }

+  private void delete(SingularityRequestCleanup requestCleanup, Iterable<SingularityTaskId> activeTaskIds){


@ssalinas I kept the name as delete since it followed the pattern set by the other actions (pause, bounce, etc.)

matush-v · 2017-01-20T23:45:10Z

Made the fixes and tested them locally. The request tasks are deleted appropriately and the request is then re-enqueued for cleanup. The user and message are preserved as well.

Also, I noticed the deploy and request history weren't updating so I fixed that. This is especially helpful when deleting a request so a user can see that it goes to a deleting state and then deleted state without refreshing the page

@ssalinas

ssalinas

let's remove those output.log and the .json files from your testing and address the two comments. Should be good to re-test in staging after that.

ssalinas · 2017-01-23T16:07:36Z

SingularityUI/app/components/requestDetail/RequestDetailPage.jsx

@@ -91,7 +91,9 @@ const mapDispatchToProps = (dispatch, ownProps) => {
    FetchRequest.trigger(ownProps.params.requestId, true),
    FetchActiveTasksForRequest.trigger(ownProps.params.requestId),
    FetchScheduledTasksForRequest.trigger(ownProps.params.requestId),
-    FetchTaskCleanups.trigger()
+    FetchTaskCleanups.trigger(),
+    FetchDeploysForRequest.trigger(ownProps.params.requestId, 5, 1),


We don't want to be fetching these two here. These are actions that will auto-refresh on an interval. Since deploy history and request history are paginated, fetching this would reset the user back to the first page on a schedule. We should instead use the then values of the buttons you are editing to make sure the correct api calls are made in reaction to a user action.

Additionally, the deploy history won't be changing in our request deletion updates, only the request history

ssalinas · 2017-01-23T16:09:05Z

SingularityUI/app/components/requestDetail/header/RequestActionButtons.jsx

@@ -162,7 +162,7 @@ const RequestActionButtons = ({requestParent, fetchRequest, fetchRequestHistory,

  const navigateAwayOnSuccess = (response) => {
    if (response.statusCode === 200) {
-      router.push('/requests');
+      then={fetchRequestAndHistory}


can move this a few lines down and get rid of the navigateAwayOnSuccess method:
<RemoveButton requestId={request.id} then={fetchRequestAndHistoryAndActiveTasks}>

Don't need the 200 check here since we are staying on the page regardless of the outcome of the api call

Added the ...AndActiveTasks portion because they will likely be updated to cleaning after the deletion is requested

ssalinas · 2017-01-24T00:43:01Z

SingularityService/src/main/java/com/hubspot/singularity/scheduler/SingularityCleaner.java

+        runBeforeKillId = Optional.of(shellRequest.getId());
+      }
+
+      taskManager.createTaskCleanup(new SingularityTaskCleanup(requestCleanup.getUser(), TaskCleanupType.USER_REQUESTED_DESTROY, start, taskId, requestCleanup.getMessage(), requestCleanup.getActionId(), runBeforeKillId));


missed this one initially, my fault here. This has some unintended consequences. With this in place, any task that is forcibly killed by a user will then cause the request to be deleted as well. We need a new TaskCleanupType that is separate. Something like REQUEST_DELETING with killLongRunningTaskInstantly and killNonLongRunningTaskInstantly both set to true like they are for the destroy

de9a21d

matush-v · 2017-02-08T20:09:40Z

Added a test to make sure that a single deleted task doesn't delete the entire request. Going to go through the branch merging (staging, qa, stable) to get it updated

ssalinas · 2017-02-10T15:54:34Z

LGTM

matush-v added 4 commits December 21, 2016 15:53

add deleting request state

91de88b

enqueue task cleanup for deleted requests

cac4dba

change request status to DELETED once all tasks are cleaned

53b397f

modularize

d2da062

ssalinas modified the milestone: 0.14.0 Jan 3, 2017

ssalinas reviewed Jan 3, 2017

View reviewed changes

matush-v added 2 commits January 9, 2017 15:10

check for null

fa02949

refactor

60c1892

matush-v commented Jan 9, 2017

View reviewed changes

updated log message

ffb5e8c

ssalinas reviewed Jan 12, 2017

View reviewed changes

matush-v added 2 commits January 12, 2017 13:33

added DELETING states and updated to correctly delete requests

3715937

update and add tests for DELETING states

b98095e

matush-v commented Jan 12, 2017

View reviewed changes

matush-v and others added 6 commits January 12, 2017 14:54

removed unwanted DELETING state

8f39f9c

preserve user and message history if it exists

4be8f90

update test for message preservation

53162de

Merge branch 'master' into deleting-state

4797793

Merge branch 'deleting-state' of github.com:HubSpot/Singularity into …

394a1e5

…deleting-state

Merge branch 'master' into deleting-state

c818b74

matush-v added 2 commits January 13, 2017 13:59

reload request page after starting a delete

a9a2dde

Merge branch 'deleting-state' of github.com:HubSpot/Singularity into …

07ff1ca

…deleting-state

reload data rather than refresh the page

05ee09e

matush-v added hs_staging labels Jan 18, 2017

matush-v added 2 commits January 20, 2017 18:40

correctly kill tasks for a delete request

b7f4f2b

update deploy and request history tables as they change

590c5c8

matush-v commented Jan 20, 2017

View reviewed changes

ssalinas requested changes Jan 23, 2017

View reviewed changes

matush-v force-pushed the deleting-state branch from 0210eac to 257dc8f Compare January 23, 2017 17:52

remove callback for delete btn

b6fff6d

matush-v force-pushed the deleting-state branch from 257dc8f to b6fff6d Compare January 23, 2017 18:09

ssalinas reviewed Jan 24, 2017

View reviewed changes

Use separate TaskCleanupType for deleting request

de9a21d

ssalinas added the hs_stable label Feb 3, 2017

test destroy task

152d7a6

matush-v merged commit cb7271f into master Feb 10, 2017

matush-v deleted the deleting-state branch February 10, 2017 15:56

tpetr removed hs_qa labels Feb 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deleting state #1386

Deleting state #1386

matush-v commented Dec 27, 2016

ssalinas Jan 3, 2017

ssalinas Jan 3, 2017

ssalinas Jan 3, 2017

matush-v Jan 9, 2017 •

edited

Loading

ssalinas commented Jan 3, 2017

matush-v Jan 9, 2017

ssalinas Jan 9, 2017

ssalinas Jan 12, 2017

matush-v Jan 12, 2017

matush-v Jan 12, 2017

matush-v Jan 12, 2017

ssalinas commented Jan 13, 2017

matush-v commented Jan 17, 2017

ssalinas commented Jan 17, 2017

matush-v Jan 20, 2017

matush-v commented Jan 20, 2017 •

edited

Loading

ssalinas left a comment

ssalinas Jan 23, 2017

ssalinas Jan 23, 2017

ssalinas Jan 24, 2017 •

edited

Loading

matush-v commented Feb 8, 2017 •

edited

Loading

ssalinas commented Feb 10, 2017

Deleting state #1386

Deleting state #1386

Conversation

matush-v commented Dec 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matush-v Jan 9, 2017 • edited Loading

Choose a reason for hiding this comment

ssalinas commented Jan 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssalinas commented Jan 13, 2017

matush-v commented Jan 17, 2017

ssalinas commented Jan 17, 2017

Choose a reason for hiding this comment

matush-v commented Jan 20, 2017 • edited Loading

ssalinas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ssalinas Jan 24, 2017 • edited Loading

Choose a reason for hiding this comment

matush-v commented Feb 8, 2017 • edited Loading

ssalinas commented Feb 10, 2017

matush-v Jan 9, 2017 •

edited

Loading

matush-v commented Jan 20, 2017 •

edited

Loading

ssalinas Jan 24, 2017 •

edited

Loading

matush-v commented Feb 8, 2017 •

edited

Loading