-
Notifications
You must be signed in to change notification settings - Fork 24.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a live streaming API? #55358
Comments
Pinging @elastic/es-search (:Search/Search) |
@jpountz When I was thinking about the changes API, one use-case that I thought for our own products was exactly the Logs application and tailing logs there. I'm curious if you've thought about this in that context as well? |
@jasontedor I have thought about it indeed. I don't think that it will be solved entirely by the Changes API because I feel like global ordering by We don't need the entire feature set of the Changes API, e.g. I don't think we would need to be informed about deletions so another option might be to use Either way we'd need something on top in order to provide global ordering by |
We discussed it today as a group. This generally felt useful, and while both
This raises interesting questions that we'll need to think about:
|
Depends on #1242 |
Thanks for considering this 🎉 While it makes total sense not to duplicate the effort for both APIs I would consider one property pretty important: It should be possible to achieve a consistent in both the changes API as well as The reason is that the latter is probably still going to be used when fetching log entries for past time intervals. |
@weltenwort The idea would be that whatever we end up exposing would take care of fetching log entries for past intervals too. The problem with |
That sounds like it would solve the search_after tiebreaker problem for us 😍 Let me know if you want to validate any API design choice in regard to the Logs UI use case early in the process. |
We'll certainly reach out when we start tackling this issue! |
Pinging @elastic/es-search (Team:Search) |
Pinging @elastic/es-search-foundations (Team:Search Foundations) |
Elasticsearch is often used to index logs and live-tailing the logs that match a given filter is a common use-case, but I think we could greatly improve the user experience here. The current approach is to periodically run a query that sorts hits by descending
@timestamp
and use a couple tricks to make these requests run efficiently.But this approach generally delivers messages out-of-order: it's likely that a request returns for the first time an event that is older than the most recent event returned by the previous request. This is mostly due to how we partition data into shards:
Would it be possible to build an API that, assuming that events get pushed to Elasticsearch in order, would be able to live-stream events in order as well?
The text was updated successfully, but these errors were encountered: