Skip to content

Commit

Permalink
Update Input Message spec to reflect the need to provide the output q…
Browse files Browse the repository at this point in the history
…ueue

Why are these changes being introduced:

- To simplify the logic in the dspace submission service app, the applications using DSS will provide the output queue
  name they expect to be reading the state back from.
- This allows DSS to only read from the input queue and write to whatever queue it is told to for errors and success.
  Since the upstream apps already need to know what queue they will read messages back from, this is not additional
  information they need to track.
- Note: this does not affect what permissions are being set anywhere. It solely allows DSS to be a bit simpler and
  changes nothing about the infrastructure requirements.

Relevant ticket(s):

- https://mitlibraries.atlassian.net/browse/ETD-435

How does this address that need:

- Updates the `MessageAttributes` spec and provides links in original spec to these new changes.
  • Loading branch information
JPrevost committed Sep 10, 2021
1 parent a9ba946 commit a448078
Show file tree
Hide file tree
Showing 2 changed files with 73 additions and 2 deletions.
8 changes: 6 additions & 2 deletions docs/architecture-decisions/0003-submission-message-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Date: 2021-08-25

Accepted

Amended by [6. Submission message spec](0006-submission-message-spec.md)

## Context

Multiple external applications will write to the SQS submit queue used by this
Expand All @@ -19,7 +21,9 @@ We will use the following submission message specification:
Each SQS Message sent to the dspace-submission-service submit queue will
contain two components, MessageAttributes and MessageBody.

#### MessageAttributes
### MessageAttributes

[Important: See Updated MessageAttributes in 6. Submission message spec](0006-submission-message-spec.md)

MessageAttributes is a JSON object containing one item, PackageID, structured
like so:
Expand All @@ -42,7 +46,7 @@ MessageAttributes = {
}
```

#### MessageBody
### MessageBody

SQS requires that the MessageBody be a string. However, this service and the
submitting applications will want to parse/create the MessageBody as JSON
Expand Down
67 changes: 67 additions & 0 deletions docs/architecture-decisions/0006-submission-message-spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# 6. Submission message spec

Date: 2021-09-09

## Status

Accepted

amends [3. Submission message spec](0003-submission-message-spec.md)

## Context

See [3. Submission message spec](0003-submission-message-spec.md) for additional context.

As we started to think about how this service will scale as more upstream applications start using it, we realized we
were putting coordination of queues responsibility into the service. For one or two apps, that is not unreasonable, but
for each new app the logistics could get unwieldy and alternate solutions are appropriate to consider.

## Decision

To simplify the logic in this DSpace Submission Service (DSS), the applications using DSS must provide the output queue
name they expect to be reading the success/errors back from.

### MessageAttributes

MessageAttributes is a JSON object containing one item, PackageID, structured
like so:

```json
MessageAttributes = {
"PackageID": {
"DataType": "String",
"StringValue": "<A unique ID created by the submitting application that
will allow said application to match the result information to each
submitted package, e.g. 'etd_123123' or '98765'. This system is agnostic about the value of the ID.>"
},
"SubmissionSource": {
"DataType": "String",
"StringValue": "<Name of the submitting system, e.g. 'ETD'. Should be
consistent for each submitting system (should not change over
time).>"

},
"OutputQueue": {
"DataType": "String",
"StringValue": "<Name of the pre-agreed upon output SQS queue to be used for the submitting system. The queue must
already exist and both systems must have appropriate access to the queue.>"
}
}
```

### MessageBody

see [3. Submission message spec](0003-submission-message-spec.md)

## Consequences

By having the submitting application be responsible for telling this service what output queue it expects to read
messages back from, it allows for the service to scale up more easily with less (or hopefully no) custom code per
submitting service.

Since the submitting application already needs to know what output queue to read from to get messages back, this does
not add any additional responsibility to the submitting app while reducing the responsibility of this service.

There is no change for the Infrastructure associated with this. All SQS queues are created and permissions assigned in
the same way they already are being managed. A submitting application cannot use an arbitrary Output queue, it needs to
have been created and permissions assigned prior to being used.

0 comments on commit a448078

Please sign in to comment.