In 1106 add docs #206

jonavellecuerdo · 2025-10-20T16:29:22Z

Purpose and background context

Update app description in README and add docs explaining the DSC workflow and the DSC CLI.

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO

What are the relevant tickets?

https://mitlibraries.atlassian.net/browse/IN-1106

jonavellecuerdo · 2025-10-20T16:31:47Z

docs/how_to_run.md

+
+_This documentation describes the DSC workflow and how to run the application._
+
+**DISCLAIMER**: While the CLI application is runnable on its own, the DSO Step Function offers a simplified user interface for running the full ETL pipeline. For more details on the DSO Step Function and how to use it, see https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/4690542593/DSpace+Submission+Orchestrator+DSO.


Will be focusing on documenting the step function in the DSO Confluence page next!

coveralls · 2025-10-20T16:33:56Z

Pull Request Test Coverage Report for Build 18690086927

Details

1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 95.234%

Totals
Change from base Build 18565553930:	0.0%
Covered Lines:	1059
Relevant Lines:	1112

💛 - Coveralls

jonavellecuerdo · 2025-10-20T16:38:15Z

docs/how_to_run.md

+  sync       Sync data between two directories using the aws s3 sync...
+```
+
+### `pipenv run dsc -w <workflow-name> -b <batch-id> create`


You will notice that when using DSC, we will need to provide the --workflow-name / -w and --batch-id / -b (CLI options even if just want to run <command-name> help). This is because we currently define the options on the main command group and set them as required=True.

We could either:

A. Decorate each command with the -w and -b CLI options and remove setting in main command group.
B. Create a shared_cli_options decorator that contains all CLI options used by the CLI commands (i.e., see example in CDPS-CURT)
C. Do nothing

@ghukill @ehanson8 Let me know what you think! This would, of course, be outside the scope of this PR.

I'm easy. Whenever we get to that point, whichever approach is feeling right works for me. I feel like everytime I really dig into click, I learn a new way to do something.

Agreed, just create a ticket with that comment and whoever picks it up can make the call

ghukill

Overall, I think it's a great start and foundation. As you mentioned, it's a living document and will likely get some updates.

Left a few comments, with the biggest couple related to how to inform readers that DSC does not actually perform ingests (or even trigger DSS directly).

docs/how_to_run.md

ghukill · 2025-10-20T17:23:43Z

docs/how_to_run.md

+It's important to note that DSC is not responsible for ingesting items into DSpace; this task is handled by _DSS_. The DSC CLI provides commands for all other steps in the DSC workflow. 
+
+### Create a batch
+DSC processes deposits in "batches", a collection of item submissions grouped by a unique identifier. DSC requires that the item submission assets (metadata and bitstream files) are uploaded to a "folder" in S3, named after the batch ID. While some requestors may upload the submission assets to S3 themselves, in other cases, these files need to be retrieved (via API requests) and uploaded during the batch creation step. 


While some requestors may upload the submission assets to S3 themselves, in other cases, these files need to be retrieved (via API requests) and uploaded during the batch creation step.

I might propose rewording this a bit. I would try and communciate that some workflows are unique in that they utilize APIs to other systems to retrieve metadata and/or bitstreams and upload them to the batch folder in S3. Try and highlight that the end result for all workflows is a batch folder in S3, but that some workflows perform some of this work programatically as part of the workflow.

Feel free to pushback, but I tripped a bit on the second part:

... in other cases, these files need to be retrieved (via API requests) and uploaded during the batch creation step

where it kind of sounds like a human user might still be involved?

See the updated description. Thanks for the discussion!

docs/how_to_run.md

ghukill · 2025-10-20T17:27:14Z

docs/how_to_run.md

+### Run DSS
+DSS consumes the submission messages from the input queue in SQS. DSS uses a client to interact with DSpace. For each item submission, DSS reads the metadata JSON file and bitstreams from S3, using the information provided in the message, and creates an item with bitstreams in DSpace.
+
+At the end of this step:
+* Result messages are written to the output queue for DSC (`dss-output-dsc`).
+
+  Note: The message is structured in accordance with the [Result Message Specification](https://github.com/MITLibraries/dspace-submission-service/blob/main/docs/specifications/result-message-specification.md).


I would propose emphasizing somewhere here that DSC does not run DSS, and thus does not perform this step.

Please see another comment in the ## The DSC Workflow section asking a related question about this section.

Updated to:

Items are ingested into DSpace by DSS

📌 Reminder: DSS is not executed by DSC and requires separate invocation.

What do you think?

ghukill · 2025-10-20T17:29:39Z

docs/how_to_run.md

+
+  Note: The message is structured in accordance with the [Result Message Specification](https://github.com/MITLibraries/dspace-submission-service/blob/main/docs/specifications/result-message-specification.md).
+
+### Inspect ingest results


Very nit picky, very subjective, totally feel free to ignore...

I might propose "Analyze" vs "Inspect". For reasons I'm having a hard time articulating, "Inspect" feels like it will then do something with what it learns, while "Analyze" feels like it has a more reporting vibe, which feels more accurate.

On a second read, "inspect" makes me think we're looking at the item ingested in DSpace, and with that, I agree "analyze" somehow feels different. 🤔

docs/how_to_run.md

ghukill · 2025-10-20T17:34:07Z

docs/how_to_run.md

+The DSC workflow consists of the following key steps:
+
+1. Create a batch
+2. Queue a batch for ingest
+3. Ingest items into DSpace


After reading the full doc, returning here.

From a DSC lens, it might be helpful to portray number 3, "Ingest items into DSpace" more as something that happens versus something DSC does?

For example, if it was "Items are ingested into DSpace by DSS", I think it would inform the language in the section below dedicated to this, which I commented on as well.

I can appreciate it's very difficult, but I think if someone were brand new to all this, they'd need a lot of help understanding that DSC does not actually perform the ingest. Once the StepFunction is more fully formed, and perhaps has it's own documentation somewhere (e.g. Confluence), it might be an opportunity in this document to link to it.

Overall, likely very little changes are probably needed here, just a bunch of little nudges that DSC doesn't do this work. It doesn't even trigger this work, that's actually DSO (StepFunction) that would do that.

Updated to:

The DSC workflow consists of the following key steps:

Create a batch

Queue a batch for ingest

Items are ingested into DSpace by DSS*

Analyze ingest results

*Important: DSC is not responsible for ingesting items into DSpace nor does it execute this process. This task is handled by DSS, which is invoked via the DSO Step Function.

What do you think?

ghukill · 2025-10-21T17:47:59Z

docs/how_to_run.md

+### Create a batch
+DSC processes deposits in "batches", a collection of item submissions grouped by a unique identifier. Generally, assets for batches are provided in one of two ways:
+
+1. Requestors upload raw metadata and bitstreams (see [How To: Batch deposits with DSO for Content Owners](https://mitlibraries.atlassian.net/wiki/spaces/INF/pages/4411326470/How-To+Batch+deposits+with+DSpace+Submission+Orchestrator+DSO+for+Content+Owners))
+2. DSC workflows retrieve raw metadata and/or bitstreams programmatically (via API requests)
+
+Once all assets are stored in a "folder" in S3, DSC verifies that metadata and bitstreams have been provided for each item submission. It is only after _all_ item submissions have been verified that DSC will establish the batch by recording each item in DynamoDB.
+
+At the end of this step:
+* If all item submission assets are complete:
+   - A batch folder with complete item submission assets exists in the DSO S3 bucket
+   - Each item submission in the batch is recorded in DynamoDB (with `status="batch_created"`)
+   - **[OPTIONAL]** An email is sent reporting the number of created item submissions. The email includes a CSV file with the batch records from DynamoDB.
+* If any item submission assets were invalid (missing metadata and/or bitstreams):
+   - A batch folder with incomplete item submission assets exists in the DSO S3 bucket
+   - **[OPTIONAL]** An email is sent reporting that zero item submissions were created. The email
+     includes a CSV file describing the failing item submissions with the corresponding error message.


:chef's kiss:

Looks great! I feel like it communicates quite a bit of complexity, pretty succintly.

ehanson8

Excellent work, very minor suggestions!

ehanson8 · 2025-10-22T18:20:13Z

docs/how_to_run.md

+Running DSC consists of the following key steps:
+
+1. Create a batch
+2. Queue a batch for ingest


Optional: I know the Step Function doc uses the word "queue" for the submit step as well but you might consider whether to just use "submit" to avoid potential confusion

Hmm, might leave as-is for now. I like that it alludes to the SQS queues. 🤔

ehanson8 · 2025-10-22T18:31:34Z

docs/how_to_run.md

+Commands:
+  create     Create a batch of item submissions.
+  finalize   Process the result messages from the DSS output queue...
+  reconcile  Reconcile bitstreams with item identifiers from the metadata.


Should this be removed or were we planning to do that later?

Good catch! When we remove the reconcile code from DSC, we can update then. :)

ehanson8 · 2025-10-22T18:33:00Z

docs/how_to_run.md

+
+Commands:
+  create     Create a batch of item submissions.
+  finalize   Process the result messages from the DSS output queue...


Excess periods here?

Ahh, this was from text-wrapping -- i.e., shows up automatically when Click prints the help docs to console~

ehanson8 · 2025-10-22T18:46:04Z

docs/how_to_run.md

+  sync       Sync data between two directories using the aws s3 sync...
+```
+
+### `pipenv run dsc -w <workflow-name> -b <batch-id> create`


Agreed, just create a ticket with that comment and whoever picks it up can make the call

jonavellecuerdo requested a review from a team as a code owner October 20, 2025 16:29

jonavellecuerdo requested review from ehanson8 and ghukill October 20, 2025 16:29

Add docs for understanding and running the DSC workflow

23806c0

jonavellecuerdo force-pushed the IN-1106-add-docs branch from 3e25eb8 to 23806c0 Compare October 20, 2025 16:31

jonavellecuerdo commented Oct 20, 2025

View reviewed changes

ghukill reviewed Oct 20, 2025

View reviewed changes

jonavellecuerdo added 2 commits October 21, 2025 10:32

Address comments in PR #206

acbe1f4

Address comments in PR #206

b3c9474

jonavellecuerdo requested a review from ghukill October 21, 2025 16:05

ghukill approved these changes Oct 21, 2025

View reviewed changes

ehanson8 approved these changes Oct 22, 2025

View reviewed changes

jonavellecuerdo merged commit 0888f6d into main Oct 22, 2025
3 checks passed


		_This documentation describes the DSC workflow and how to run the application._

		DISCLAIMER: While the CLI application is runnable on its own, the DSO Step Function offers a simplified user interface for running the full ETL pipeline. For more details on the DSO Step Function and how to use it, see https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/4690542593/DSpace+Submission+Orchestrator+DSO.


		Note: The message is structured in accordance with the [Result Message Specification](https://github.com/MITLibraries/dspace-submission-service/blob/main/docs/specifications/result-message-specification.md).

		### Inspect ingest results

In 1106 add docs #206

In 1106 add docs #206

Uh oh!

Conversation

jonavellecuerdo commented Oct 20, 2025

Purpose and background context

Includes new or updated dependencies?

Changes expectations for external applications?

What are the relevant tickets?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18690086927

Details

💛 - Coveralls

Uh oh!

jonavellecuerdo Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghukill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Items are ingested into DSpace by DSS

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonavellecuerdo Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehanson8 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

coveralls commented Oct 20, 2025 •

edited

Loading

jonavellecuerdo Oct 20, 2025 •

edited

Loading

jonavellecuerdo Oct 21, 2025 •

edited

Loading