Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specifying index names in request body #61141

Open
jloleysens opened this issue Aug 14, 2020 · 6 comments
Open

Allow specifying index names in request body #61141

jloleysens opened this issue Aug 14, 2020 · 6 comments
Labels
:Core/Infra/REST API REST infrastructure and utilities >enhancement help wanted adoptme Team:Core/Infra Meta label for core/infra team

Comments

@jloleysens
Copy link
Contributor

jloleysens commented Aug 14, 2020

Many ES APIs currently provide a way to query for a set of indices by specifying comma separated index patterns. In certain scenarios, scripted querying of these endpoints can create very long URIs (+4096 bytes) that trigger an ES exception (elastic/kibana#72616, https://discuss.elastic.co/t/url-too-long-too-many-indices/143593/4) or could run into a web server's URI length limit in certain configurations (although these are typically longer).

This is constrained to scenarios where:

  1. You want a very specific subset of indices on the fly (aliases, index patterns or data streams probably cannot help)
  2. The subset are many such that the names CSV'd (probably URI encoded too) are more than 4096 bytes
  3. You are probably interacting with ES in scripted programmatic way to achieve specific functionality (i.e., this is a latent issue that will be triggered once the number of indices has grown large)

It seems the index names would need to passed through in some other way (perhaps a PUT or GET with a body?) or creating and deleting aliases on the fly although I am not sure that would have been viable in a case like elastic/kibana#72616, but fortunately the new resolve endpoint could be used to get the information that was needed.

Is there a general way of setting things up so that this limit will not be hit given the current APIs?

@jloleysens jloleysens added the needs:triage Requires assignment of a team area label label Aug 14, 2020
@DaveCTurner
Copy link
Contributor

#52367 is a more specific instance of this. Today the answer is no, but being able to specify a longer list of indices in a channel other than the URI (maybe the request body) seems like a reasonable thing to want IMO.

@DaveCTurner DaveCTurner added the :Core/Infra/REST API REST infrastructure and utilities label Aug 14, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/REST API)

@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Aug 14, 2020
@DaveCTurner DaveCTurner added team-discuss and removed needs:triage Requires assignment of a team area label labels Aug 14, 2020
@pugnascotia
Copy link
Contributor

ES could also be behind a proxy that also restricts URI length, so again it would be useful to have another option for specifying indices.

@pugnascotia
Copy link
Contributor

We talked about this in the Core/Infra sync. We felt that it is reasonable to be able to specify the list of indices in the request body as well as the URL. The next step here would be to work out the scope and impact of the required changes, since we'd need to update the REST API in a number of places.

@rjernst rjernst changed the title [Question] Interacting with clusters with many indices (long URIs) Allow specifying index names in request body Sep 15, 2020
@rjernst
Copy link
Member

rjernst commented Sep 15, 2020

As noted above, we don't object to such a change, but this is also an edge case that most users don't come anywhere near to hitting. So I have marked this as an adoptme. Please feel free to open a PR.

@rjernst rjernst added the needs:triage Requires assignment of a team area label label Dec 3, 2020
@jaymode jaymode removed the needs:triage Requires assignment of a team area label label Dec 9, 2020
@i-plusplus
Copy link
Contributor

@rjernst @pugnascotia Can I take it up ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/REST API REST infrastructure and utilities >enhancement help wanted adoptme Team:Core/Infra Meta label for core/infra team
Projects
None yet
Development

No branches or pull requests

7 participants