diff --git a/docs-docfx/articles/core/formats/clonable-format.md b/docs-docfx/articles/core/formats/clonable-format.md new file mode 100644 index 00000000..69e644be --- /dev/null +++ b/docs-docfx/articles/core/formats/clonable-format.md @@ -0,0 +1,26 @@ +# Cloneable format + +.NET does not provide an interface to guarantee a +[deep clone](https://learn.microsoft.com/en-us/dotnet/api/system.icloneable?view=net-7.0#remarks) +implementation. + +The [`ICloneableFormat`](xref:Yarhl.FileFormat.ICloneableFormat) gives the +possibility to a format implementation to specify how it should _deep_ clone its +data into a new format. This could be a simple as copying its properties into a +new object or in the case of binary data copying all its bytes into a new +stream. + +[!code-csharp[cloneable](./../../../../src/Yarhl.Examples/Formats.cs?name=CloneableFormat)] + +The interface already implements `IFormat` so it's not needed to implement both. + +> [!NOTE] +> This interface is not required to be implemented by every format but some APIs +> of the library relies on it. For instance it's only possible to clone a +> [node via its constructor]() +> if it has a format that implements +> [`ICloneableFormat`](xref:Yarhl.FileFormat.ICloneableFormat). + +> [!NOTE] +> The built-in formats from _Yarhl_ implements +> [`ICloneableFormat`](xref:Yarhl.FileFormat.ICloneableFormat). diff --git a/docs-docfx/articles/core/formats/converters-usecases.md b/docs-docfx/articles/core/formats/converters-usecases.md new file mode 100644 index 00000000..25e67de2 --- /dev/null +++ b/docs-docfx/articles/core/formats/converters-usecases.md @@ -0,0 +1,208 @@ +# Advanced uses cases for converters + +The [converters](./converters.md) topic covers the standard use cases _convert +one format into another_. However often you may run into more advanced +scenarios. The following sections tries to provide some architecture guidance. + +## Convert multiple formats into one + +The recommended architecture is to create a converter for the **main format and +use parameters\*** to pass the additional formats. + +Let's try to see with a couple of examples: + +### Create a font file from an image file and an JSON file + +We identify the _main_ format as the JSON structure as it contains most of the +information required to setup the format. An image file goes as parameter to be +used for the glyphs of the font. + +> [!NOTE] +> Instead of passing a JSON binary data, pre-convert it already into its +> structure / class. It will simplify the implementation of the converter and it +> could be it can be re-used for more cases (e.g. in the future you decide to +> change to YAML). + +```csharp +public class Font2Binary : IConverter +{ + private readonly Image fontImage; + + public Font2Binary(Image fontImage) + { + this.fontImage = fontImage; + } + + public BinaryFormat Convert(FontFormat source) + { + // use the two objects to serialize into the target font format. + } +} +``` + +### Convert an indexed image with a palette into an RGB image + +The _main_ format would be the indexed image as contains more information +representing the target format. A palette is required to transform the pixel +indexes into a final RGB color. + +```csharp +public class IndexedImage2RgbImage : IConverter +{ + private readonly Palette palette; + + public IndexedImage2RgbImage(Palette palette) + { + this.palette = palette; + } + + public RgbImage Convert(IndexedImage source) + { + // convert each pixel into RGB using the provided palette. + } +} +``` + +### Additional patterns for many to one + +We described other uses cases that may fit some use cases. In our experience +they don't work as good as the previous mentioned _parameter_ approach. + +#### Intermediary types + +Create an intermediary type that groups all the required formats to convert. For +instance you could create a class `IndexedImageWithPalette`, put inside the two +objects and create a converter for +`IConverter`. + +This may simplify your converter but it can create more complex APIs. Now users +will need to _convert_ their formats into this intermediary representation to +use the converter. + +It may prevent a _fluent-like_ usage of the converters when used with the +[node](../virtual-file-system/nodes.md) APIs. It won't allow to convert one +_node_ passing other _node_ as parameters. + +#### Using tuples as input type + +This similar to the above case. It has the further limitation that it can't +evolve over the time. If you need an additional format or parameter in the +future you will breaking the API for the users making it a bit more messy. + +## Convert one format into many + +Depending on the use cases you may want to: + +1. **Convert the format into a container type `NodeContainerFormat`** that + contains a child per output format. +2. Create a **separate** converter for each target format. + +> [!TIP] +> Check-out the [container](../virtual-file-system/nodes.md) topic to learn more +> about containers. + +### Convert an RGB image into indexed image and palette + +Reverse operation from +[convert an indexed image with palette into RGB image](#convert-an-indexed-image-with-a-palette-into-an-rgb-image). +As this converter will generate a palette where the _indexed pixels_ will point +to, we will need to return it as well. + +We will return a container with a child `image` and another `palette`. + +```csharp +public class RgbImage2IndexedImage : IConverter +{ + public NodeContainerFormat Convert(RgbImage source) + { + // Run a quantization algorithm that generates a palette and indexed pixels + var container = new NodeContainerFormat(); + + var indexedImageNode = new Node("image", indexedImage); + container.Root.Add(indexedImageNode); + + var paletteNode = new Node("palette", palette); + container.Root.Add(paletteNode); + + return container; + } +} +``` + +The user of the API would be able to extract both formats later: + +```csharp +using Node imageNode = NodeFactory.FromFile("image.png", FileOpenMode.Read) + .TransformWith() + .TransformWith(); + +var indexedImage = imageNode.Children["image"].GetFormatAs(); +var palette = imageNode.Children["palette"].GetFormatAs(); +``` + +### Export a font into information and image + +In this case it could be a better approach to separate the converters: + +1. A `Font2BinaryInfo` converter that serializes the charset map and other + information into JSON / YAML. +2. A `Font2Image` converter that exports the glyphs into an image. + +Each converter runs a different process to generate the output. These two output +formats are not generated at the same time (as it was the case above). + +By splitting it allows users to run the one they need when they need it. It may +not be required to generate an image all the time or vice-versa. + +## Convert multiple formats into many + +This use case would be covered by the two previous cases: combining converting +[multiple formats into one](#convert-multiple-formats-into-one) and +[one format into many](#convert-one-format-into-many). + +## Updating / Importing data in a format + +Sometimes you may run a process that modifies existing data of a format +**without creating a new format**. + +For instance, if there is an unknown or complex binary format like an executable +and we want to **only change its text**. + +In these cases we can create a converter that **returns the same input +instance** after processing. We can pass the data to import as a **parameter**. + +Let's see an example: + +```csharp +public record TextBlockInfo(uint Position, string Text); + +public class ExecutableTextImporter : IConverter +{ + private readonly IEnumerable textInfos; + + public ExecutableTextImporter(IEnumerable textInfos) + { + this.textInfos = textInfos ?? throw new ArgumentNullException(nameof(textInfos)); + } + + public IBinary Convert(IBinary source) + { + var writer = new DataWriter(source.Stream); + foreach (var info in textInfos) { + writer.Stream.Position = info.Position; + + // you should check it doesn't overwrite more data than it can + writer.Write(info.Text); + } + + return source; + } +} +``` + +> [!TIP] +> It could be a good idea to create a new `BinaryFormat` to copy the input +> before overwriting data. In that case you would be returning a **new binary +> format** but with the existing content. In this way you don't modify the +> existing file on disk but create a new one in case something wrong happens and +> you want to run it again. diff --git a/docs-docfx/articles/core/formats/converters.md b/docs-docfx/articles/core/formats/converters.md index 4e127236..54375ff5 100644 --- a/docs-docfx/articles/core/formats/converters.md +++ b/docs-docfx/articles/core/formats/converters.md @@ -1,26 +1,77 @@ # Converters -You can convert a model (formats) into another with _converter_ classes. A -_Yarhl converter_ implements the interface +You can convert a [formats](./formats.md) (model) into another format by using a +_converter_ class. A _Yarhl converter_ implements the interface [`IConverter`](xref:Yarhl.FileFormat.IConverter`2) and provides the method [`TDst Convert(TSrc)`](). +This method creates a new object in the target type _converting_ the data from +the input. For instance the converter [`Po2Binary`](xref:Yarhl.Media.Text.Po2Binary) -implements `IConverter` allowing you to convert a +implements `IConverter`. It allows to convert a [`Po`](xref:Yarhl.Media.Text.Po) model format into a [_binary_ format](xref:Yarhl.IO.BinaryFormat). This is also known as -**serialization**. +**serialization**. You can later write this binary data into a file on disk. -You can use it by creating a new instance and calling its `Convert(Po)` method: +In a similar way, the converter [`Binary2Po`](xref:Yarhl.Media.Text.Binary2Po) +implements `IConverter` to convert binary data into a +[`Po`](xref:Yarhl.Media.Text.Po) model (also known as _reading_ or +_deserializing_). + +We could have more conversions between formats. For instance +`IConverter` or `IConverter`. This is sometimes referred +as _exporting_ and _importing_ formats. _Converters_ simplify all these +operations by their common denominator: **converting models.** + +Let's see how to _serialize_ / convert a _Po_ model into binary data to write on +disk: [!code-csharp[serialize PO](./../../../../src/Yarhl.Examples/Converters.cs?name=SerializePo)] ## Implementing a new converter -TODO +To create a new converter, create a new class and implement the interface +[``](xref:Yarhl.FileFormat.IConverter`2). `TSrc` is the +type (or base type / interface) you are going to convert into a new object of +`TDst` type. + +> [!NOTE] +> It is possible to have a class implementing more than one converter at a type. +> However this can be confusing for the user. Our recommendation is that each +> class implements only one +> [`IConverter`](xref:Yarhl.FileFormat.IConverter`2) interface. For +> instance, create `Po2Binary` and `Binary2Po` instead of just `Binary2Po` +> having the two implementations. + +As an example, let's implement a new converter that reads binary data and +creates a [container type](../virtual-file-system/nodes.md) (like a file +system). + +First we create a new class for our converter: `BinaryArchive2Container` to do +the operation _binary data_ -> _container class_ (deserializing). + +```csharp +public class BinaryArchive2Container : IConverter +{ + // TODO: Implement interface. +} +``` + +Now let's add the required method `Convert` for the interface. + +```csharp +public NodeContainerFormat Convert(IBinary source) +{ + var container = new NodeContainerFormat(); + // TODO: do something with the source data. + return container; +} +``` + +Finally let's read some data to fill the container. This example binary format +contains a set of binary files inside. ```csharp -// Implement a new format container from binary (file) into a virtual file system. public class BinaryArchive2Container : IConverter { public NodeContainerFormat Convert(IBinary source) @@ -45,32 +96,93 @@ public class BinaryArchive2Container : IConverter return container; } } +``` + +And voilà. To use our new converter we just need to create a new instance and +pass some binary data. +```csharp // Convert the binary file into a virtual folder (no disk writing). -using Node root = NodeFactory.FromFile("file.bin", FileOpenMode.Read); -root.TransformWith(); // Binary -> node format +var fileStream = DataStreamFactory.FromFile("myArchive.bin", FileOpenMode.Read); +using var binaryFormat = new BinaryFormat(); + +var binary2Container = new BinaryArchive2Container(); +using var container = binary2Container.Convert(binaryFormat); -// Extract a child into disk. -Node child = root.Children["text.json"] // Navigate the children -child.Stream.WriteTo("output/text.json"); // Export to the disk (creates missing dirs) +// Now we can inspect or extract the content of the container +Node child = container.Children["text.json"] +child.Stream.WriteTo(child.Name); ``` -## `IConverter` interface +## Parameters -TODO +Frequently your converter may require additional parameters than just the input +object to do the conversion. For instance in a compressor you may need to ask +your users to provide the level of compression to do. Or you may need to know +the line ending for a text format. You may need to know if the target CPU is big +or little endian or the text encoding. -## Converters with parameters +In any of these cases, you can ask the user to provide this required or optional +information in the constructor of the converter class. -TODO +> [!TIP] +> If your converter can run with some _default_ parameters, provide a +> parameter-less constructor to simplify its usage for common use cases. -## Converting many formats into one +```csharp +public class RgbImage2IndexedImage : IConverter +{ + private readonly IColorQuantization quantization; -TODO + // Parameter-less constructors for a default value that can be used in most cases. + public RgbImage2IndexedImage() + { + quantization = new ColorQuantization(); + } -## Converting one format into many + // Allow the user to customize the converter to their needs. + public RgbImage2IndexedImage(IColorQuantization customQuantization) + { + quantization = customQuantization; + } -TODO + public IndexedImage Converter(RgbImage source) + { + // Use the quantization instance to convert RGB colors into an indexed image + // ... + } +} +``` -## Updating / Importing data in a format +## `IConverter` interface -TODO +> [!IMPORTANT] +> Normally the [`IConverter`](xref:Yarhl.FileFormat.IConverter) (no generics +> version) is for internal use only. Unless writing a new framework or generic +> tools, use always +> [`IConverter`](xref:Yarhl.FileFormat.IConverter`2). + +You may notice that there is also an +[`IConverter`](xref:Yarhl.FileFormat.IConverter) interface that takes no +generics. The [`IConverter`](xref:Yarhl.FileFormat.IConverter`2) +_implements_ this base interface. + +This is an empty interface used only internally to enforce some basic +type-safety when due to technical reason we can't know the types of the +converter, so we can't use +[`IConverter`](xref:Yarhl.FileFormat.IConverter`2). + +For instance +[`Node.TransformWith(IConverter converter)`]() +uses the base interface to provide a simple API. Requiring the fully typed +interface would make users to specify to repeat the types: +`node.TransformWith(myConverter)` as the compiler +cannot guess these types at compile-type. By having the simple interface we can +just use `node.TransformWith(myConverter)`. + +Note that when the API uses [`IConverter`](xref:Yarhl.FileFormat.IConverter) it +will run reflection run-time checks to ensure the argument is valid. It will +check that the variable or type implements +[`IConverter`](xref:Yarhl.FileFormat.IConverter`2) and that the +input object is valid for this type. Although it may hit some nanoseconds of +performance, it provides better error messages. diff --git a/docs-docfx/articles/core/formats/formats.md b/docs-docfx/articles/core/formats/formats.md index e9822001..043b79cf 100644 --- a/docs-docfx/articles/core/formats/formats.md +++ b/docs-docfx/articles/core/formats/formats.md @@ -72,46 +72,20 @@ convert. You can theoretically implement `IConverter`. However, in order to provide some features the library expects that every format implements the _empty_ interface [`IFormat`](xref:Yarhl.FileFormat.IFormat). -By using the `IFormat` interface it allows the APIs to: +By using the [`IFormat`](xref:Yarhl.FileFormat.IFormat) interface it allows the +APIs to: - Provide extension methods that applies to formats only (like `ConvertWith`). - Provide type discovery for _formats_ via _Yarhl.Plugins_. - Prevent unboxing performance issues. -### Working with existing models +## Working with existing models -Models should implement the `IFormat` interface. If you have a model and cannot -be modified to inherit from the interface, then it's possible to create a -_format wrapper_. +Models should implement the [`IFormat`](xref:Yarhl.FileFormat.IFormat) +interface. If you have a model and cannot be modified to inherit from the +interface, then it's possible to create a _format wrapper_. For instance, let's see how to provide a format-compatible class for a third-party sound format `ThirdPartyWave`: [!code-csharp[format wrapper](./../../../../src/Yarhl.Examples/Formats.cs?name=FormatWrapper)] - -## Cloneable formats - -.NET does not provide an interface to guarantee a -[deep clone](https://learn.microsoft.com/en-us/dotnet/api/system.icloneable?view=net-7.0#remarks) -implementation. - -The [`ICloneableFormat`](xref:Yarhl.FileFormat.ICloneableFormat) gives the -possibility to a format implementation to specify how it should _deep_ clone its -data into a new format. This could be a simple as copying its properties into a -new object or in the case of binary data copying all its bytes into a new -stream. - -[!code-csharp[cloneable](./../../../../src/Yarhl.Examples/Formats.cs?name=CloneableFormat)] - -The interface already implements `IFormat` so it's not needed to implement both. - -> [!NOTE] -> This interface is not required to be implemented by every format but some APIs -> of the library relies on it. For instance it's only possible to clone a -> [node via its constructor]() -> if it has a format that implements -> [`ICloneableFormat`](xref:Yarhl.FileFormat.ICloneableFormat). - -> [!NOTE] -> The built-in formats from _Yarhl_ implements -> [`ICloneableFormat`](xref:Yarhl.FileFormat.ICloneableFormat). diff --git a/docs-docfx/articles/core/getting-started/architecture.md b/docs-docfx/articles/core/getting-started/architecture.md index e6fbb44c..b2a4c305 100644 --- a/docs-docfx/articles/core/getting-started/architecture.md +++ b/docs-docfx/articles/core/getting-started/architecture.md @@ -1,3 +1,19 @@ # Framework architecture +TODO: base goal and requirements + +## Reading, deserializing, exporting + +TODO + +## Core library and plugins + +TODO + +## Related tools + +TODO + +## End game + TODO diff --git a/docs-docfx/articles/core/toc.yml b/docs-docfx/articles/core/toc.yml index 47ec48c2..b5e4e5b0 100644 --- a/docs-docfx/articles/core/toc.yml +++ b/docs-docfx/articles/core/toc.yml @@ -11,6 +11,12 @@ href: ./formats/formats.md - name: Converters href: ./formats/converters.md +- name: Advanced + items: + - name: Clonable format + href: ./formats/clonable-format.md + - name: Use cases for converters + href: ./formats/converters-usecases.md - name: 📁 Virtual file system - name: Node overview diff --git a/src/Yarhl.Examples/Converters.cs b/src/Yarhl.Examples/Converters.cs index fe7bcfa3..6196ae42 100644 --- a/src/Yarhl.Examples/Converters.cs +++ b/src/Yarhl.Examples/Converters.cs @@ -32,10 +32,12 @@ public void SerializePo() // Create a new converter instance var po2binary = new Po2Binary(); + + // Convert! using var binaryPoFormat = po2binary.Convert(poFormat); - // Binary format is a wrapper over a DataStream (enhanced Stream) - // we can now save the Stream into a file + // Binary format is a wrapper over a DataStream (enhanced System.IO.Stream) + // We can now save the Stream into a file binaryPoFormat.Stream.WriteTo("strings.po"); #endregion }