Issues with empty PollForDecisionTask and PollForActivityTask responses

[mfine]

Running into issues with empty polls where de-serialization fails because SWF doesn't seem to be conforming to its models:

Failing

Empty PollForDecisionTask (FAIL)

[Client Request] {
  host      = swf.us-west-2.amazonaws.com:443
  secure    = True
  method    = POST
  target    = Just SimpleWorkflowService.PollForDecisionTask
  timeout   = Just 70000000000000
  redirects = 0
  path      = /
  query     =
  headers   = host: swf.us-west-2.amazonaws.com; x-amz-date: 20151202T185851Z; x-amz-content-sha256: xxx; x-amz-target: SimpleWorkflowService.PollForDecisionTask; content-type: application/x-amz-json-1.0; authorization: AWS4-HMAC-SHA256 Credential=xxx, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=xxx
  body      = {"domain":"hello-world","reverseOrder":true,"taskList":{"name":"hello-world-queue"},"maximumPageSize":100}
}
[Client Response] {
  status  = 200 OK
  headers = x-amzn-requestid: xxx; content-type: application/x-amz-json-1.0; content-length: 47
}
[Raw Response Body] {
{"previousStartedEventId":0,"startedEventId":0}
}
[SerializeError] {
  service = SWF
  status  = 200 OK
  message = key "taskToken" not present
}

Empty PollForActivityTask (FAIL)

[Client Request] {
  host      = swf.us-west-2.amazonaws.com:443
  secure    = True
  method    = POST
  target    = Just SimpleWorkflowService.PollForActivityTask
  timeout   = Just 70000000000000
  redirects = 0
  path      = /
  query     =
  headers   = host: swf.us-west-2.amazonaws.com; x-amz-date: 20151202T190200Z; x-amz-content-sha256: xxx; x-amz-target: SimpleWorkflowService.PollForActivityTask; content-type: application/x-amz-json-1.0; authorization: AWS4-HMAC-SHA256 Credential=xxx, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=xxx
  body      = {"domain":"hello-world","taskList":{"name":"hello-queue"}}
}
[Client Response] {
  status  = 200 OK
  headers = x-amzn-requestid: xxx; content-type: application/x-amz-json-1.0; content-length: 20
}
[Raw Response Body] {
{"startedEventId":0}
}
[SerializeError] {
  service = SWF
  status  = 200 OK
  message = key "taskToken" not present
}

Passing

Non-Empty PollForDecisionTask (PASS)

[Client Request] {
  host      = swf.us-west-2.amazonaws.com:443
  secure    = True
  method    = POST
  target    = Just SimpleWorkflowService.PollForDecisionTask
  timeout   = Just 70000000000000
  redirects = 0
  path      = /
  query     =
  headers   = host: swf.us-west-2.amazonaws.com; x-amz-date: 20151202T190829Z; x-amz-content-sha256: xxx; x-amz-target: SimpleWorkflowService.PollForDecisionTask; content-type: application/x-amz-json-1.0; authorization: AWS4-HMAC-SHA256 Credential=xxx, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=xxx
  body      = {"domain":"hello-world","reverseOrder":true,"taskList":{"name":"hello-world-queue"},"maximumPageSize":100}
}
[Client Response] {
  status  = 200 OK
  headers = x-amzn-requestid: xxx; content-type: application/x-amz-json-1.0; content-length: 2609
}
[Raw Response Body] {
{"events":[{"decisionTaskStartedEventAttributes":{"scheduledEventId":8},"eventId":9,"eventTimestamp":1.449083309828E9,"eventType":"DecisionTaskStarted"},{"decisionTaskScheduledEventAttributes":{"startToCloseTimeout":"60","taskList":{"name":"hello-world-queue"}},"eventId":8,"eventTimestamp":1.449083237789E9,"eventType":"DecisionTaskScheduled"},{"activityTaskTimedOutEventAttributes":{"scheduledEventId":5,"startedEventId":6,"timeoutType":"START_TO_CLOSE"},"eventId":7,"eventTimestamp":1.449083237789E9,"eventType":"ActivityTaskTimedOut"},{"activityTaskStartedEventAttributes":{"scheduledEventId":5},"eventId":6,"eventTimestamp":1.449083137778E9,"eventType":"ActivityTaskStarted"},{"activityTaskScheduledEventAttributes":{"activityId":"2e99c751-002a-44eb-8340-a65445d4216d","activityType":{"name":"Hello","version":"1.0"},"decisionTaskCompletedEventId":4,"heartbeatTimeout":"NONE","input":"{\n  \"step-0\": \"execute\"\n}\n","scheduleToCloseTimeout":"NONE","scheduleToStartTimeout":"60","startToCloseTimeout":"100","taskList":{"name":"hello-queue"}},"eventId":5,"eventTimestamp":1.449083137731E9,"eventType":"ActivityTaskScheduled"},{"decisionTaskCompletedEventAttributes":{"scheduledEventId":2,"startedEventId":3},"eventId":4,"eventTimestamp":1.449083137731E9,"eventType":"DecisionTaskCompleted"},{"decisionTaskStartedEventAttributes":{"scheduledEventId":2},"eventId":3,"eventTimestamp":1.449083137612E9,"eventType":"DecisionTaskStarted"},{"decisionTaskScheduledEventAttributes":{"startToCloseTimeout":"60","taskList":{"name":"hello-world-queue"}},"eventId":2,"eventTimestamp":1.449083115695E9,"eventType":"DecisionTaskScheduled"},{"eventId":1,"eventTimestamp":1.449083115695E9,"eventType":"WorkflowExecutionStarted","workflowExecutionStartedEventAttributes":{"childPolicy":"ABANDON","executionStartToCloseTimeout":"300","input":"{\n  \"step-0\": \"execute\"\n}\n","parentInitiatedEventId":0,"taskList":{"name":"hello-world-queue"},"taskStartToCloseTimeout":"60","workflowType":{"name":"HelloWorld","version":"1.0"}}}],"previousStartedEventId":3,"startedEventId":9,"taskToken":"AAAAKgAAAAIAAAAAAAAAAxLGr2/uDoZMAhWy1gmiF7C+/OXh8EZDgs/BFrE74YfGXtmlfgC2OmnceXa8bhjhQpl2fG1jJovu82ED7GLbKRTmzfCS8YppGU+RbBVr9nlKKjYa/ToB/AjM8RW5BeMN+9Rr/e70C0bJOGs68r7UewhJzAWYuxPQfY2iPO1JgeqEVGdL8SknhmrYT8MOciTnZX7CmTuKRbQ0wX4ZjL0LlKqQTsAqOvVheC+iaIOf92rDfijkL5EQYxvgojIdsRWHJJR2zeWoslZwvveNbY8i3b51FD1p/dxB+4ouUt9BurXBPFkj67X0j45p9xkwwRDYkw==","workflowExecution":{"runId":"23IvIkYB9xFvVJwMpM+0ANpM7rEjD31QqX5do6138ujAA=","workflowId":"dc8df543-69b7-411e-b059-974a20e98b96"},"workflowType":{"name":"HelloWorld","version":"1.0"}}
}

Non-Empty PollForActivityTask (PASS)

[Client Request] {
  host      = swf.us-west-2.amazonaws.com:443
  secure    = True
  method    = POST
  target    = Just SimpleWorkflowService.PollForActivityTask
  timeout   = Just 70000000000000
  redirects = 0
  path      = /
  query     =
  headers   = host: swf.us-west-2.amazonaws.com; x-amz-date: 20151202T190917Z; x-amz-content-sha256: xxx; x-amz-target: SimpleWorkflowService.PollForActivityTask; content-type: application/x-amz-json-1.0; authorization: AWS4-HMAC-SHA256 Credential=xxx, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-target, Signature=xxx
  body      = {"domain":"hello-world","taskList":{"name":"hello-queue"}}
}
[Client Response] {
  status  = 200 OK
  headers = x-amzn-requestid: xxx; content-type: application/x-amz-json-1.0; content-length: 655
}
[Raw Response Body] {
{"activityId":"76d0006e-35d7-4c87-adfd-0f1bc31fc901","activityType":{"name":"Hello","version":"1.0"},"input":"{\n  \"step-0\": \"execute\"\n}\n","startedEventId":12,"taskToken":"AAAAKgAAAAIAAAAAAAAAA9L0rkHPWDPNCx40dFjaOO3dVqwdyypqjpOK4XCyYW94qw/L3KPJAivnmnn24j69XEes94iu1uQQ7p8dgppErUl9H7rsvrxLmoF5EhlE/WJLj1Mw/TxpN8NqaunH6XtoKu7URZkcth2D2IkmDzPVRhsFdyPutP2n5JpRh3ThOgjcynEeyMnxOHBx8XGMjNi6n6VwUsg3Bi10N2XVL6JrDzT7CiK44HR/2lxAGR3n+/eLHG1P50yANjpDLKTbbzridEsFphZ5lJpJ9mujjTI6OrFQU841ymocnwe/1OgUcusGhGmnlVoKotoChIcB6MwFBg==","workflowExecution":{"runId":"23IvIkYB9xFvVJwMpM+0ANpM7rEjD31QqX5do6138ujAA=","workflowId":"dc8df543-69b7-411e-b059-974a20e98b96"}}
}

I'm confused because the models for both indicate these are mandatory attributes:

https://github.com/brendanhay/amazonka/blob/develop/gen/model/swf/2012-01-25/service-2.json#L812-L818
https://github.com/brendanhay/amazonka/blob/develop/gen/model/swf/2012-01-25/service-2.json#L1745-L1753

Looking around in other related SDKs (e.g., aws-sdk-go), there's mention of the fields coming back as nil (which I think is how their JSON parsers will work in the absence of a field). It's almost like maybe there are no required fields - everything should be Maybe :(

[mfine]

Sample from the Java SDK:

https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-swf-libraries/src/main/java/com/amazonaws/services/simpleworkflow/flow/worker/DecisionTaskPoller.java#L177-L179

[mfine]

Workaround I'm using for now is to wrap the polling:

checkSerializeError :: MonadXXX m => Error -> m ()
checkSerializeError = \case
  e@(SerializeError s) -> do
    unless check $ throwM e where
      check =
        s ^. serializeStatus  == ok200 &&
        s ^. serializeAbbrev  == "SWF" &&
        s ^. serializeMessage == "key \"taskToken\" not present"
  e -> throwM e

xxx :: MonadXXX m => m ()
xxx =
  handle checkSerializeError $ do
    poll stuff

But yuck.

[brendanhay]

Great, thanks for going the extra mile with the details @mfine. I'll get it fixed shortly.

[brendanhay]

And by shortly I mean two weeks! Apologies for the belated fix. Please have a look at #260 and let me know if that should cover both cases.

I'll update the tests/properties separately to ensure regression is avoided in the future.

[mfine]

Thanks! Nice solution - sorry it had to come to that :( Will check things out!