v1.13.1 - Hosted adjuncts, `maxWidth` and `openFullMax`

Summary

• maxWidth and openFullMax
  • Orchestrator respects maxWidth and openFullMax, see ADR 0010 and ADR 0011
  • Orchestrator strictly parses IIIF ImageApi 2.1 and 3.0 requests.
  • Orchestrator will always refresh cached asset before generating info.json.
  • Display maxWidth and openFullMax in Portal
• Support hosted adjuncts, updates to API/Orchestrator/Engine, see https://dlcs.github.io/public-docs/api-doc/adjuncts/
• Introduce adjunct batches
• Handling of 'stub' assets. Space 0 repurposed, see RFC 022
• Delete CustomerStorage record when space deleted.

Bugfixes

• Catch edge-case where not-found image returns incorrect status code #1197
• Handle s3 objects larger than 50GB #1200

What's Changed

• Hosted Adjunct - API Management by @p-kaczynski in #1102
• Set missing ingesting/error prop on adjunct hydra output model by @p-kaczynski in #1108
• Add openFullMax and maxWidth (API only) by @donaldgray in #1105
• Update ThumbCreator to use maxUnauth replacements by @donaldgray in #1107
• Bump to localstack:4.13 by @donaldgray in #1109
• Update docker compose readme by @donaldgray in #1101
• Replace maxUnauthorised in Orchestrator by @donaldgray in #1116
• Refresh OrchestrationImage info.json create by @donaldgray in #1119
• Add maxWidth and openFullMax to portal by @donaldgray in #1120
• Engine handles hosted adjuncts by @p-kaczynski in #1115
• PullRequest template by @donaldgray in #1126
• Add InlineAnnotation to the API by @JackLewis-digirati in #1118
• Add support to orchestrator for inline annotations by @JackLewis-digirati in #1124
• Add /version endpoint to API and Orchestrator by @donaldgray in #1131
• Handle very small maxWidth values and "sizes" in info.json by @donaldgray in #1130
• Initial RFC for stub assets by @donaldgray in #1129
• Bugfixes for inline annotation work by @JackLewis-digirati in #1139
• Orchestrator serves adjuncts by @p-kaczynski in #1136
• Use orchestrator url for hosted adjuncts id in manifest builder by @p-kaczynski in #1146
• Add ability to delete hosted adjuncts by @JackLewis-digirati in #1138
• Fix hosted adjunct being updated getting deleted by @JackLewis-digirati in #1162
• Add concurrency to github action by @donaldgray in #1156
• Support ?include=adjuncts query parameter by @donaldgray in #1165
• Add provides to adjunct by @JackLewis-digirati in #1168
• Allow POSTing of multiple Adjuncts in one request for creation by @p-kaczynski in #1167
• Implement bulk delete of adjuncts by @JackLewis-digirati in #1169
• Immediate completion of none channel by @JackLewis-digirati in #1170
• Create batch of adjuncts by @donaldgray in #1171
• Publish notification AdjunctBatch completion by @donaldgray in #1172
• Add GET adjunct batch endpoint by @donaldgray in #1173
• Specify actual type when serialising completion by @donaldgray in #1177
• Always return single adjunct on PUT by @donaldgray in #1175
• Include 'type' in batch completion notifications by @donaldgray in #1179
• Prevent invalid adjunct ids in deletion by @donaldgray in #1181
• Support ?include on POST allImages by @donaldgray in #1185
• UpdateHandlers abort if item not in DB by @donaldgray in #1178
• Set certain fields to optional for the 'None' channel by @JackLewis-digirati in #1187
• Exclude query-parameters in generated adjunct ids by @donaldgray in #1190
• Allow for stub assets to be added to space 0 by @JackLewis-digirati in #1191
• Allow create batch of images to work with a minimal payload in the stub asset space by @JackLewis-digirati in #1193
• Generated manifests include adjuncts for 'none' by @donaldgray in #1196
• Improve OrchestrationImage caching - avoid unnecessary proxying by @donaldgray in #1198
• Handle bucket copy for files greater than 50GB by @donaldgray in #1202
• Validate "annotations" adjuncts are JSON by @donaldgray in #1204
• Use null for the global space storage record by @JackLewis-digirati in #1203
• Update wording of adjunct error message by @JackLewis-digirati in #1205
• Disallow spaces that aren't a positive integer by @JackLewis-digirati in #1206
• RFC: Hosted Annotation Adjuncts id Property Handling by @p-kaczynski in #1176
• Allow modifiying the id of a hosted annotation id by @JackLewis-digirati in #1209
• Decrement image storage for adjuncts by @JackLewis-digirati in #1211
• Delete CustomerStorage on space delete by @donaldgray in #1213
• Text-services ADR by @donaldgray in #1210
• Update main branch by @donaldgray in #1214

Warning

When deploying to an existing environment, the MaxWidth appSetting value for API, Engine and Orchestrator may need to be set depending on existing image-server configuration. The backing image-server config may need to be set to allow this MaxWidth, or disabled to allow orchestrator to control.

OldestAllowedInfoJson will also need to be set to ensure info.json files are refreshed. This may result in an increase in Orchestration operations.

Warning

Contains migrations

• New columns in "Images" and UPDATE statement. This could be slow / expensive in some environments! #1105
• Adjunct-related #1115, #1118, #1153, #1171
• Repurposing space0 #1191, #1203

Note

This PR requires infrastructure changes.

• Required:
  • SQS queues for adjunct updating and deletion
  • SNS topic for adjunct modification
• Optional: AdjunctBatch completion SNS topic. If present API or Engine will publish a notification when an AdjunctBatch is completed. If missing a warning will be logged.
• Optional: For environments where larger AV files are expected, timebased queue visibility_timeout could be increased to allow for handling. In practice this won't have much affect as messages are only retried once.

Note

This PR introduces configuration changes.

Service	AppSetting	Required?	Description	Default
CleanupHandler	AWS:SQS:AdjunctDeleteNotificationQueueName	N	Delete notification queue name	null
CleanupHandler	AWS:SQS:AdjunctUpdateNotificationQueueName	N	Update notification queue name	null
API	AWS:SNS:AdjunctModifiedNotificationTopicArn	N	Update notification topic	null
Engine	AWS:SNS:AdjunctBatchCompletedTopicArn	N	ARN for AdjunctBatch Completion SNS topic	null
API	AWS:SNS:AdjunctBatchCompletedTopicArn	N	ARN for AdjunctBatch Completion SNS topic	null
Engine	AWS:S3:CopyPartConcurrency	N	Number of concurrent part uploads	4
Orchestrator	MaxAdjunctSizeBytes	N	Max permitted annotation adjunct size	5 * 1024 * 1024

Full Changelog: v1.12.1...v1.13.1