Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce the sub-entry publish support #4

Closed
Gsantomaggio opened this issue Sep 22, 2021 · 5 comments
Closed

Introduce the sub-entry publish support #4

Gsantomaggio opened this issue Sep 22, 2021 · 5 comments

Comments

@Gsantomaggio
Copy link
Collaborator

Gsantomaggio commented Sep 22, 2021

it would be useful to have the sub-entry publish support.

see: https://rabbitmq.github.io/rabbitmq-stream-java-client/stable/htmlsingle/#producer-sub-entry-size-configuration-entry

The internals:
https://github.com/rabbitmq/osiris/blob/HEAD/src/osiris_log.erl#L163-L263

@Gsantomaggio
Copy link
Collaborator Author

If want to know more detail about, please let me know.

@qweeze
Copy link
Owner

qweeze commented Sep 22, 2021

Thanks! I will look into it in the next few days

@Gsantomaggio
Copy link
Collaborator Author

In Java and Go have all the compressions type directly on the client:

NONE = byte(0)
GZIP   = byte(1)
SNAPPY = byte(2)
LZ4    = byte(3)
ZSTD   = byte(4)

In the .NET client, we decided to implement only GZIP and NONE then add the possibility to use external plugins to use:

SNAPPY = byte(2)
LZ4    = byte(3)
ZSTD   = byte(4)

I'd follow the same .NET policy

@Gsantomaggio
Copy link
Collaborator Author

To make it easy I would do as .NET like a send overload:

var subEntryMessages = List<Messages>();
for (var i = 1; i <= 500; i++)
{
    var message = new Message(Encoding.UTF8.GetBytes($"SubBatchMessage_{i}"));
    subEntryMessages.Add(message);
}

await producer.Send(1, subEntryMessages, CompressionType.Gzip);
messages.Clear();

So in python it would be:

publish(
        self,
        stream: str,
        message[] <--- array or list of messages
       compression_type

Note that for 1 publishing id you can have 1 or more messages.
publishing_id 1 ---> [] messages

@DanielePalaia
Copy link
Collaborator

Closing this one as already implemented in: #54

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants