.Net: Add overload ctor to ImageContent that takes an string "url"

[dersia]

When using the Vision models of AzureOpenAI and OpenAI, the API allows for an Image to be attached either as an URL or as a Data URI. In Semantic Kernel we do have ImageContent as a KernelContent-Component, but unfortunately, it takes only an System.Uri as the parameter for the object.

I suggest to following addition to ImageContent:

Proposed API

public sealed class ImageContent : KernelContent
{
    // existing, I think we should just change the type
    // this is a breaking change, although it don't think we should have both
    // but alternatively we could just add another Property 'DataUri' instead of changing the type
    public Uri? Uri { get; set; }
    // replaced by this:
    public string Uri { get; set; }

    // existing, though changed to call the new overload instead of base
    public ImageContent(
        Uri uri,
        string? modelId = null,
        object? innerContent = null,
        Encoding? encoding = null,
        IReadOnlyDictionary<string, object?>? metadata = null)
        : this(uri.ToString(), modelId, innerContent, encoding, metadata)
    {
        this.Uri = uri;
    }

    public ImageContent(
        string dataUri,
        string? modelId = null,
        object? innerContent = null,
        Encoding? encoding = null,
        IReadOnlyDictionary<string, object?>? metadata = null)
        : base(innerContent, modelId, metadata)
    {
        this.Uri = uri;
    }
}

This would allow us to use Data URIs with embedded images for the Vision APIs.

Usage

var chatHistory = new ChatHistory();
var image = System.Convert.ToBase64String(imageBytes);
var imageDataLink = $"data:{imageContentType};base64,{image}";
chatHistory.Add(new ImageContent(imageDataLink));
var result = await chat.GetChatMessageContentAsync(chatHistory);

Alternate Design

This would be a breaking Change, since we change the Type of the Uri property on ImageContent, so instead

• we could add another property DataUri to ImageContent, but this would also mean that we have to check which of the two properties are used and prefer one other the other, if both are filled.
• We could also just make a private string field that holds the DataUri and when the Uri ctor is used we just store the Uri as string in that field and only use the field on implementation site.
• We could also make ImageContent take a ROS, byte[] or even just a Stream of the image and handle all the Base64Encoding in the ImageContent ctor (I think I would prefer a static Create method over ctor for this option, sonce we can than read the stream async), but this would mean, that we also need a second parameter with the image-ContentType and this would not allow for non base64 encoded data uris

I am happy to work on the implementation, since I have to do it anyway, because we need the solution fairly soon.

[hez2010]

Another way is to remove the length limitation of Uri in .NET: dotnet/runtime#96544

data:image/jpeg;base64,... is a valid uri and can be parsed by System.Uri without issue, but the .NET implementation of Uri has a length limitation (65520 bytes) so it can only process small images.

[dersia]

Another way is to remove the length limitation of Uri in .NET: dotnet/runtime#96544

data:image/jpeg;base64,... is a valid uri and can be parsed by System.Uri without issue, but the .NET implementation of Uri has a length limitation (65520 bytes) so it can only process small images.

this was my initial thought, but I am not sure how this would work with validation that is done today in System.Uri and if the validation needs to be changed, than this might be a breaking change.
if dotnet/runtime#96544 is a valid option, then I would also suggest to extend the UriKind enum to also include Data.

[Krzysztof318]

I think this is bad idea to introduce this breaking change in abstraction layer.
So better would be create additional property.

[dersia]

I think this is bad idea to introduce this breaking change in abstraction layer. So better would be create additional property.

I would also prefer one of the alternative designs, when I think more about it. I still think it is a too bad, that Uri is a set-able public property :/

[dmytrostruk]

@dersia This is exactly the reason why ImageContent.Uri is nullable at the moment - to avoid breaking changes for this class in the future. In OpenAI API, there is image_url.url object and it's not clear what data it accepts now and what possibly can be accepted in the future:

But from Semantic Kernel point of view, we want developers to use Uri in case if it's URL to image located on some server, and we want new properties like string? Base64 to pass image in base64 format or string Path to provide path to local image file (we should not add this, just as an example).

So, answering your question - we don't want to introduce breaking changes and new property that keeps image as base64 should work. Hope that makes sense, thank you!

[dersia]

@dersia This is exactly the reason why ImageContent.Uri is nullable at the moment - to avoid breaking changes for this class in the future. In OpenAI API, there is image_url.url object and it's not clear what data it accepts now and what possibly can be accepted in the future:

But from Semantic Kernel point of view, we want developers to use Uri in case if it's URL to image located on some server, and we want new properties like string? Base64 to pass image in base64 format or string Path to provide path to local image file (we should not add this, just as an example).

So, answering your question - we don't want to introduce breaking changes and new property that keeps image as base64 should work. Hope that makes sense, thank you!

@dmytrostruk sounds good to me. I have started on the implementation (as I said, we need this sooner rather than later) and in this implementation I have gone down the non breaking path. I also added overloads for byte[] and ROS<byte>. I am just adding new tests for this and then I could create a pr if you'd like.

also I was wondering if there would be a need for a stream version that would use a crypto stream to base64 encode on the fly while writing to the stream. this would be interesting for larger image files, but I don't think this is a concern as of now, right?

[dmytrostruk]

I also added overloads for byte[] and ROS<byte>

Did you consider using System.Memory.Data.BinaryData? It supports ReadOnlyMemory<byte>, Stream, string and byte[].

I am just adding new tests for this and then I could create a pr if you'd like.

That would be awesome, thank you!

also I was wondering if there would be a need for a stream version that would use a crypto stream to base64 encode on the fly while writing to the stream. this would be interesting for larger image files, but I don't think this is a concern as of now, right?

Yes, I think it's not a concern, at least for now.

[dersia]

I also added overloads for byte[] and ROS<byte>

Did you consider using System.Memory.Data.BinaryData? It supports ReadOnlyMemory<byte>, Stream, string and byte[].

I am just adding new tests for this and then I could create a pr if you'd like.

That would be awesome, thank you!

also I was wondering if there would be a need for a stream version that would use a crypto stream to base64 encode on the fly while writing to the stream. this would be interesting for larger image files, but I don't think this is a concern as of now, right?

Yes, I think it's not a concern, at least for now.

System.Memory.Data.BinaryData is a great idea! I didn't think of it. unfortunately the Stream ctor for BinaryData still isn't good enough for large files, but since that isn't a concern for now, I will just implement it with BinaryData .

I will also adjust the tests for to work with BinaryData.

[dersia]

@dmytrostruk please find my PR #4919

[dersia]

Thanks for merging this @RogerBarreto, however I thought we would wait for Azure/azure-rest-api-specs#27780 to be done so we can fully integrate this. Because as of now I does not do anything :/