Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change keys method to use Stream #797

Merged
merged 3 commits into from
Feb 3, 2023

Conversation

thomastaylor312
Copy link
Contributor

This is a follow up to #792 (I'll rebase once that is merged). Please see this comment thread for additional details around the discussion of changing this API: #792 (comment)

@Jarema Jarema changed the title ref(kv): Changes keys method to use Stream Fix kv key listing and properly filter deleted and purged keys Jan 20, 2023
@Jarema
Copy link
Member

Jarema commented Jan 31, 2023

Getting back to this:

I think keys() should not return the Error twice - on initial call and on each next().

If we want to have iterator over keys (which we do), we always will have error on subscription, but we can defer errors from calling keys() to iterating.

@caspervonb WDYT?

@thomastaylor312
Copy link
Contributor Author

@Jarema like I noted in the comment in the other PR, the double error thing happens all over the place in the KV bucket code and in other parts of the jetstream code too.

So if we don't want to return a double error, then we should change it everywhere for consistency. However, I think it is ok to do have the double error as each one indicates something entirely different. The first error says "I couldn't even set up this Stream type for you at all. An error in the stream indicates that getting a single item failed, but not the whole stream

@caspervonb
Copy link
Collaborator

If we want to have iterator over keys (which we do), we always will have error on subscription, but we can defer errors from calling keys() to iterating.

@caspervonb WDYT?

Different classes of errors imo, just that we haven't made them concrete at this time so the API doesn't reflect that very well.
If we're going to do a stream then the errors should be streaming as they happen imo.

/// # Ok(())
/// # }
/// ```
pub async fn keys(&self) -> Result<impl futures::Stream<Item = Result<String, Error>>, Error> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe wrap this in its own type (Keys) for consistency, can just have an inner History and have poll_next implement the filtering.

Suggested change
pub async fn keys(&self) -> Result<impl futures::Stream<Item = Result<String, Error>>, Error> {
pub async fn keys(&self) -> Result<Keys, Error> {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think I started it and it was a lot of extra code. Can definitely go back and add it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now done

keys.insert(entry.key);
}
}
self.stream.delete_consumer(&consumer_name).await?;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that this is no longer required. The server was cleaning up the consumer just fine, but the bug fixed in #824 was actually causing the problem with duplicate consumers

@thomastaylor312 thomastaylor312 changed the title Fix kv key listing and properly filter deleted and purged keys ref(kv): Changes keys method to use Stream Feb 2, 2023
@thomastaylor312
Copy link
Contributor Author

@caspervonb I created the new Keys stream, so I think this should be ready to go

This is an optional commit that I figured I'd do while I was here.
The original documentation for this method suggested that it should
be returning a `Stream` rather than an iter. So I implemented a
stream version of it if desired. This commit can be popped off (or
moved to a separate PR) depending on maintainer preference

Signed-off-by: Taylor Thomas <taylor@cosmonic.com>
Copy link
Collaborator

@caspervonb caspervonb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last looks @Jarema?

lgtm

Copy link
Member

@Jarema Jarema left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Jarema Jarema changed the title ref(kv): Changes keys method to use Stream Change keys method to use Stream Feb 3, 2023
@Jarema Jarema merged commit 70b5caa into nats-io:main Feb 3, 2023
@thomastaylor312 thomastaylor312 deleted the ref/keys_stream branch February 19, 2023 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants