Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get image and thumbnail channels to work in engine #777

Merged
merged 17 commits into from
Apr 9, 2024
Merged

Get image and thumbnail channels to work in engine #777

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1c87f82
initial commit
JackLewis-digirati Mar 26, 2024
22e36ae
test fixes
JackLewis-digirati Mar 27, 2024
981500f
adding more tests
JackLewis-digirati Mar 28, 2024
48228a0
removing unnecessary fields
JackLewis-digirati Apr 2, 2024
b9ec448
fixing tests and updating to use KnownDeliveryChannelPolicies
JackLewis-digirati Apr 2, 2024
931d889
changes before code review
JackLewis-digirati Apr 2, 2024
d97f9b5
using system agnostic file separator
JackLewis-digirati Apr 2, 2024
1686c16
adding transient thumbs to storagge-keys
JackLewis-digirati Apr 2, 2024
5fb0c88
updates to use a transient key
JackLewis-digirati Apr 3, 2024
2f10c69
update to s3-ambient + no image channel to not be saved in DLCS
JackLewis-digirati Apr 3, 2024
dad2e71
fixing tests
JackLewis-digirati Apr 3, 2024
1b6920d
updating to use none channel
JackLewis-digirati Apr 3, 2024
8b37c0d
adding comments + removing unnecessary parameter
JackLewis-digirati Apr 4, 2024
f401c40
code review fixes
JackLewis-digirati Apr 8, 2024
d3e13ce
final changes
JackLewis-digirati Apr 8, 2024
f7e4df5
code review changes
JackLewis-digirati Apr 8, 2024
4693671
Update ImageServerClient to pass thumb dir to CantaloupeThumbsClient
donaldgray Apr 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 17 additions & 15 deletions docs/storage-keys.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,23 @@ The DLCS uses a number of S3 keys in various buckets to store assets. These gene
> The default 'storage-key' is `$"{customer}/{space}/{assetKey}"`.
> The various bucketnames below equate to those in `S3Settings`

| Name | Format | Example | Description |
| ------------------ | ------------------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------- |
| Storage | `"{StorageBucket}/{storage-key}"` | `dlcs-storage/1/2/foo` | Default location where generated derivatives are stored |
| Storage Original | `"{StorageBucket}/{storage-key}/original"` | `dlcs-storage/1/2/foo/original` | Where direct-copy of origin is stored. For `/file/` delivery or images with `use-original` policy |
| InfoJson | `"{StorageBucket}/{storage-key}/info/{image-server}/{version}/info.json"` | `dlcs-storage/1/2/foo/info/cantaloupe/v3/info.json` | Location where pregenerated info.json stored |
| Audio output | `"{StorageBucket}/{storage-key}/full/max/default.{extension}"` | `dlcs-storage/1/2/foo/full/max/default.mp3` | Location where transcoded audio stored |
| Video output | `"{StorageBucket}/{storage-key}/full/full/max/max/0/default.{extension}"` | `dlcs-storage/1/2/foo/full/full/max/max/0/default.mp4` | Location where transcoded video stored |
| Timebased Metadata | `"{StorageBucket}/{storage-key}/metadata"` | `dlcs-storage/1/2/foo/metadata` | XML blob storing ElasticTranscoder JobId |
| Thumbnail | `"{ThumbsBucket}/{storage-key}/{access}/{longestEdge}.jpg"` | `dlcs-thumbs/1/2/foo/open/100.jpg` | Location of specific thumbnail |
| Legacy Thumbnail | `"{ThumbsBucket}/{storage-key}/full/{w},{h}/0/default.jpg"` | `dlcs-thumbs/1/2/foo/full/100,200/0/default.jpg` | Location of specific thumbnail using legacy layout |
| Thumbnail Sizes | `"{ThumbsBucket}/{storage-key}/s.json"` | `dlcs-thumbs/1/2/foo/s.json` | JSON blob storing knowng thumbnails |
| Largest Thumbnail | `"{ThumbsBucket}/{storage-key}/low.jpg"` | `dlcs-thumbs/1/2/foo/low.jpg` | The location of the largest generated thumbnail |
| Thumbnail Root | `"{ThumbsBucket}/{storage-key}/"` | `dlcs-thumbs/1/2/foo/` | Root key where thumbnails will reside |
| Output Location | `"{OutputBucket}/{storage-key}/"` | `dlcs-output/1/2/foo/` | Root key where DLCS 'output' is stored (e.g. projected NQ to PDF or Zip) |
| Origin Location | `"{OriginBucket}/{storage-key}"` | `dlcs-origin/1/2/foo` | Location where directly uploaded bytes are stored |
| Name | Format | Example | Description |
| --------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------- |
| Storage | `"{StorageBucket}/{storage-key}"` | `dlcs-storage/1/2/foo` | Default location where generated derivatives are stored |
| Storage Original | `"{StorageBucket}/{storage-key}/original"` | `dlcs-storage/1/2/foo/original` | Where direct-copy of origin is stored. For `/file/` delivery or images with `use-original` policy |
| InfoJson | `"{StorageBucket}/{storage-key}/info/{image-server}/{version}/info.json"` | `dlcs-storage/1/2/foo/info/cantaloupe/v3/info.json` | Location where pregenerated info.json stored |
| Audio output | `"{StorageBucket}/{storage-key}/full/max/default.{extension}"` | `dlcs-storage/1/2/foo/full/max/default.mp3` | Location where transcoded audio stored |
| Video output | `"{StorageBucket}/{storage-key}/full/full/max/max/0/default.{extension}"` | `dlcs-storage/1/2/foo/full/full/max/max/0/default.mp4` | Location where transcoded video stored |
| Timebased Metadata | `"{StorageBucket}/{storage-key}/metadata"` | `dlcs-storage/1/2/foo/metadata` | XML blob storing ElasticTranscoder JobId |
| Thumbnail | `"{ThumbsBucket}/{storage-key}/{access}/{longestEdge}.jpg"` | `dlcs-thumbs/1/2/foo/open/100.jpg` | Location of specific thumbnail |
| Legacy Thumbnail | `"{ThumbsBucket}/{storage-key}/full/{w},{h}/0/default.jpg"` | `dlcs-thumbs/1/2/foo/full/100,200/0/default.jpg` | Location of specific thumbnail using legacy layout |
| Thumbnail Sizes | `"{ThumbsBucket}/{storage-key}/s.json"` | `dlcs-thumbs/1/2/foo/s.json` | JSON blob storing known thumbnails |
| Largest Thumbnail | `"{ThumbsBucket}/{storage-key}/low.jpg"` | `dlcs-thumbs/1/2/foo/low.jpg` | The location of the largest generated thumbnail |
| Thumbnail Root | `"{ThumbsBucket}/{storage-key}/"` | `dlcs-thumbs/1/2/foo/` | Root key where thumbnails will reside |
| Output Location | `"{OutputBucket}/{storage-key}/"` | `dlcs-output/1/2/foo/` | Root key where DLCS 'output' is stored (e.g. projected NQ to PDF or Zip) |
| Origin Location | `"{OriginBucket}/{storage-key}"` | `dlcs-origin/1/2/foo` | Location where directly uploaded bytes are stored |
| Transient Images | `"{StorageBucket}/stransient/{storage-key}"` | `dlcs-thumbs/1/2/foo/open/100.jpg` | Location of transient images, that will be cleaned up by lifecycle policies |
JackLewis-digirati marked this conversation as resolved.
Show resolved Hide resolved


## Timebased

Expand Down
7 changes: 7 additions & 0 deletions src/protagonist/DLCS.AWS/S3/IStorageKeyGenerator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,4 +166,11 @@ public interface IStorageKeyGenerator
/// </summary>
/// <returns><see cref="ObjectInBucket"/> for JSON object containing credentials for a user's origin strategy</returns>
ObjectInBucket GetOriginStrategyCredentialsLocation(int customerId, string originStrategyId);

/// <summary>
/// Get <see cref="ObjectInBucket"/> root location for the origin bucket
JackLewis-digirati marked this conversation as resolved.
Show resolved Hide resolved
/// </summary>
/// <param name="assetId">asset id the request is for</param>
/// <returns><see cref="ObjectInBucket"/> for specified asset's metadata file</returns>
RegionalisedObjectInBucket GetTransientImageLocation(AssetId assetId);
}
6 changes: 6 additions & 0 deletions src/protagonist/DLCS.AWS/S3/S3StorageKeyGenerator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,4 +191,10 @@ public ObjectInBucket GetOriginStrategyCredentialsLocation(int customerId, strin
var key = $"{customerId}/origin-strategy/{originStrategyId}/credentials.json";
return new ObjectInBucket(s3Options.SecurityObjectsBucket, key);
}

public RegionalisedObjectInBucket GetTransientImageLocation(AssetId assetId)
{
var key = GetStorageKey(assetId);
return new RegionalisedObjectInBucket(s3Options.StorageBucket, $"transient/{key}", awsSettings.Region);
}
}
8 changes: 8 additions & 0 deletions src/protagonist/DLCS.Core/FileSystem/FileSystem.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace DLCS.Core.FileSystem;

Expand Down Expand Up @@ -45,4 +47,10 @@ public long GetFileSize(string path)
}

public void SetLastWriteTimeUtc(string path, DateTime dateTime) => File.SetLastWriteTimeUtc(path, dateTime);

public async Task CreateFileFromStream(string path, Stream stream, CancellationToken cancellationToken = default)
{
await using var fileStream = new FileStream(path, FileMode.Create);
await stream.CopyToAsync(fileStream, cancellationToken);
}
}
4 changes: 4 additions & 0 deletions src/protagonist/DLCS.Core/FileSystem/IFileSystem.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace DLCS.Core.FileSystem;

Expand All @@ -14,4 +17,5 @@ public interface IFileSystem
bool FileExists(string path);
long GetFileSize(string path);
void SetLastWriteTimeUtc(string path, DateTime dateTime);
Task CreateFileFromStream(string path, Stream stream, CancellationToken cancellationToken = default);
}
134 changes: 90 additions & 44 deletions src/protagonist/DLCS.Model.Tests/Assets/AssetXTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,23 @@ public class AssetXTests
public void GetAvailableThumbSizes_IncludeUnavailable_Correct_MaxUnauthorisedNoRoles()
{
// Arrange
var thumbnailPolicy = new ThumbnailPolicy
{
Id = "TestPolicy",
Name = "TestPolicy",
Sizes = "800,400,200,100"
var asset = new Asset {Width = 5000, Height = 2500, MaxUnauthorised = 500,
ImageDeliveryChannels = new List<ImageDeliveryChannel>
{
new()
{
DeliveryChannelPolicyId = KnownDeliveryChannelPolicies.ImageDefault,
Channel = AssetDeliveryChannels.Thumbnails,
DeliveryChannelPolicy = new DeliveryChannelPolicy
{
PolicyData = "[\"800,800\",\"400,400\",\"200,200\",\"100,100\"]"
}
}
}
};

var asset = new Asset {Width = 5000, Height = 2500, MaxUnauthorised = 500};

// Act
var sizes = asset.GetAvailableThumbSizes(thumbnailPolicy, out var maxDimensions, true);
var sizes = asset.GetAvailableThumbSizes(out var maxDimensions, true);

// Assert
sizes.Should().BeEquivalentTo(new List<Size>
Expand All @@ -43,17 +49,25 @@ public void GetAvailableThumbSizes_IncludeUnavailable_Correct_MaxUnauthorisedNoR
public void GetAvailableThumbSizes_NotIncludeUnavailable_Correct_MaxUnauthorisedNoRoles()
{
// Arrange
var thumbnailPolicy = new ThumbnailPolicy
var asset = new Asset
{
Id = "TestPolicy",
Name = "TestPolicy",
Sizes = "800,400,200,100"
Width = 5000, Height = 2500, MaxUnauthorised = 500,
ImageDeliveryChannels = new List<ImageDeliveryChannel>
{
new()
{
DeliveryChannelPolicyId = KnownDeliveryChannelPolicies.ImageDefault,
Channel = AssetDeliveryChannels.Thumbnails,
DeliveryChannelPolicy = new DeliveryChannelPolicy
{
PolicyData = "[\"800,800\",\"400,400\",\"200,200\",\"100,100\"]"
}
}
}
};

var asset = new Asset {Width = 5000, Height = 2500, MaxUnauthorised = 500};

// Act
var sizes = asset.GetAvailableThumbSizes(thumbnailPolicy, out var maxDimensions, false);
var sizes = asset.GetAvailableThumbSizes(out var maxDimensions, false);

// Assert
sizes.Should().BeEquivalentTo(new List<Size>
Expand All @@ -71,17 +85,25 @@ public void GetAvailableThumbSizes_NotIncludeUnavailable_Correct_MaxUnauthorised
public void GetAvailableThumbSizes_IncludeUnavailable_Correct_IfRolesNoMaxUnauthorised()
{
// Arrange
var thumbnailPolicy = new ThumbnailPolicy
var asset = new Asset
{
Id = "TestPolicy",
Name = "TestPolicy",
Sizes = "800,400,200,100",
Width = 5000, Height = 2500, Roles = "GoodGuys", MaxUnauthorised = -1,
ImageDeliveryChannels = new List<ImageDeliveryChannel>
{
new()
{
DeliveryChannelPolicyId = KnownDeliveryChannelPolicies.ImageDefault,
Channel = AssetDeliveryChannels.Thumbnails,
DeliveryChannelPolicy = new DeliveryChannelPolicy
{
PolicyData = "[\"800,800\",\"400,400\",\"200,200\",\"100,100\"]"
}
}
}
};

var asset = new Asset {Width = 5000, Height = 2500, Roles = "GoodGuys", MaxUnauthorised = -1};

// Act
var sizes = asset.GetAvailableThumbSizes(thumbnailPolicy, out var maxDimensions, true);
var sizes = asset.GetAvailableThumbSizes(out var maxDimensions, true);

// Assert
sizes.Should().BeEquivalentTo(new List<Size>
Expand Down Expand Up @@ -110,7 +132,7 @@ public void GetAvailableThumbSizes_NotIncludeUnavailable_Correct_IfRolesNoMaxUna
var asset = new Asset {Width = 5000, Height = 2500, Roles = "GoodGuys", MaxUnauthorised = -1};

// Act
var sizes = asset.GetAvailableThumbSizes(thumbnailPolicy, out var maxDimensions, false);
var sizes = asset.GetAvailableThumbSizes(out var maxDimensions, false);

// Assert
sizes.Should().BeNullOrEmpty();
Expand All @@ -123,17 +145,25 @@ public void GetAvailableThumbSizes_NotIncludeUnavailable_Correct_IfRolesNoMaxUna
public void GetAvailableThumbSizes_RestrictsAvailableSizes_IfHasRolesAndMaxUnauthorised()
{
// Arrange
var thumbnailPolicy = new ThumbnailPolicy
var asset = new Asset
{
Id = "TestPolicy",
Name = "TestPolicy",
Sizes = "800,400,200,100",
Width = 2500, Height = 5000, Roles = "GoodGuys", MaxUnauthorised = 399,
ImageDeliveryChannels = new List<ImageDeliveryChannel>
{
new()
{
DeliveryChannelPolicyId = KnownDeliveryChannelPolicies.ImageDefault,
Channel = AssetDeliveryChannels.Thumbnails,
DeliveryChannelPolicy = new DeliveryChannelPolicy
{
PolicyData = "[\"800,800\",\"400,400\",\"200,200\",\"100,100\"]"
}
}
}
};

var asset = new Asset {Width = 2500, Height = 5000, Roles = "GoodGuys", MaxUnauthorised = 399};

// Act
var sizes = asset.GetAvailableThumbSizes(thumbnailPolicy, out var maxDimensions);
var sizes = asset.GetAvailableThumbSizes(out var maxDimensions);

// Assert
sizes.Should().BeEquivalentTo(new List<Size>
Expand All @@ -150,17 +180,25 @@ public void GetAvailableThumbSizes_RestrictsAvailableSizes_IfHasRolesAndMaxUnaut
public void GetAvailableThumbSizes_ReturnsAvailableAndUnavailableSizes_ButReturnsMaxDimensionsOfAvailableOnly_IfIncludeUnavailable()
{
// Arrange
var thumbnailPolicy = new ThumbnailPolicy
var asset = new Asset
{
Id = "TestPolicy",
Name = "TestPolicy",
Sizes = "800,400,200,100",
Width = 2500, Height = 5000, Roles = "GoodGuys", MaxUnauthorised = 399,
ImageDeliveryChannels = new List<ImageDeliveryChannel>
{
new()
{
DeliveryChannelPolicyId = KnownDeliveryChannelPolicies.ImageDefault,
Channel = AssetDeliveryChannels.Thumbnails,
DeliveryChannelPolicy = new DeliveryChannelPolicy
{
PolicyData = "[\"800,800\",\"400,400\",\"200,200\",\"100,100\"]"
}
}
}
};

var asset = new Asset {Width = 2500, Height = 5000, Roles = "GoodGuys", MaxUnauthorised = 399};

// Act
var sizes = asset.GetAvailableThumbSizes(thumbnailPolicy, out var maxDimensions, true);
var sizes = asset.GetAvailableThumbSizes(out var maxDimensions, true);

// Assert
sizes.Should().BeEquivalentTo(new List<Size>
Expand All @@ -179,17 +217,25 @@ public void GetAvailableThumbSizes_ReturnsAvailableAndUnavailableSizes_ButReturn
public void GetAvailableThumbSizes_HandlesImageBeingSmallerThanThumbnail()
{
// Arrange
var thumbnailPolicy = new ThumbnailPolicy
var asset = new Asset
{
Id = "TestPolicy",
Name = "TestPolicy",
Sizes = "800,400,200,100"
Width = 300, Height = 150,
ImageDeliveryChannels = new List<ImageDeliveryChannel>
{
new()
{
DeliveryChannelPolicyId = KnownDeliveryChannelPolicies.ImageDefault,
Channel = AssetDeliveryChannels.Thumbnails,
DeliveryChannelPolicy = new DeliveryChannelPolicy
{
PolicyData = "[\"800,800\",\"400,400\",\"200,200\",\"100,100\"]"
}
}
}
};

var asset = new Asset { Width = 300, Height = 150 };

// Act
var sizes = asset.GetAvailableThumbSizes(thumbnailPolicy, out var maxDimensions, true);
var sizes = asset.GetAvailableThumbSizes(out var maxDimensions, true);

// Assert
sizes.Should().BeEquivalentTo(new List<Size>
Expand Down
Loading
Loading