New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with index name handling #2058

Closed
bernd opened this Issue Apr 12, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@bernd
Member

bernd commented Apr 12, 2016

Problem description

In 1728772 we introduced a stricter pattern to check if an index is handled by Graylog to avoid touching anything else.

I see two problems with that:

  1. The "System/Indices" page still shows other indices. It looks like the pattern is not used when requesting the list of indices.
  2. In some cases we actually need to handle indices that do not match the pattern. For example the upcoming archive plugin needs to be able to restore indices with a different name to avoid any name clashes.

Trying to open the details for the graylog_17_restored_archive does not work.

wsrjtnkaaaaaelftksuqmcc

The details do not show up, the spinner keeps spinning.

axhqnlxtfxlqaaaaaelftksuqmcc

In the developer tools I see a failed request for the archive because the name does not match the strict pattern.

{
  "type":"ApiError",
  "message":"Index [graylog_17_restored_archive] doesn't look like an index managed by Graylog."
}

Steps to reproduce the problem

  1. Use archive plugin in 2.0.0-beta.2 to create an archive for an index.
  2. Restore the archive index.
  3. Navigate to "System/Indices"
  4. See an index named like *_restored_archive
  5. Try to click "Show Details / Actions"
  6. Spinner does not go away and see errors in the developer console.

Solution Discussion

To avoid showing the non-^graylog_[0-9]+$ indices on the "System/Indices" page, we could just make sure to apply the strict pattern for the index list as well.

The problem is that the archive plugin cannot be used for restores at the moment because of the strict pattern handling. An archived index can be restored but then there will be no index range for it and we also cannot delete it.

One idea would be to make the index name patterns pluggable to allow plugins to inject a new name format.

Any other or better ideas?

Environment

  • Graylog Version: 2.0.0-beta.3-SNAPSHOT (882c097)
  • Elasticsearch Version: 2.3.0

@bernd bernd added the bug label Apr 12, 2016

@bernd bernd added this to the 2.0.0 milestone Apr 12, 2016

@kroepke

This comment has been minimized.

Member

kroepke commented Apr 12, 2016

I'd be fine with hard coding the archive plugin name format for now and properly fixing this in 2.1.

Personally I consider the index naming non-public API and I'm hesitant to open that up to plugins without a good plan. Especially once we want to allow multiple indices, possibly per stream or other criteria, this will quickly become messy.

How about allowing a suffix of _restored_archive for getting information and recalc of index ranges but disallowing them to be part of the "which is the latest index" logic?

@joschi

This comment has been minimized.

Contributor

joschi commented Apr 12, 2016

How about allowing a suffix of _restored_archive for getting information and recalc of index ranges but disallowing them to be part of the "which is the latest index" logic?

I'd also prefer that as a quick solution while thinking about a proper solution (maybe also incorporating solutions which do not use Elasticsearch) in a later version.

bernd added a commit that referenced this issue Apr 12, 2016

Improve handling of Elasticsearch indices
- Make sure to only return Graylog indices in the indices and indices
  overview resources.
- Until we find a better solution, hard code the index suffix for restored
  archive indices to make it possible to search in restored indices and
  close/delete them.

Fixes #2058
@bernd

This comment has been minimized.

Member

bernd commented Apr 12, 2016

Thanks for the feedback! I created a PR: #2062

bernd added a commit that referenced this issue Apr 13, 2016

Improve handling of Elasticsearch indices
- Make sure to only return Graylog indices in the indices and indices
  overview resources.
- Until we find a better solution, hard code the index suffix for restored
  archive indices to make it possible to search in restored indices and
  close/delete them.

Fixes #2058

@joschi joschi closed this in #2062 Apr 13, 2016

joschi added a commit that referenced this issue Apr 13, 2016

Improve handling of Elasticsearch indices (#2062)
Make sure to only return Graylog indices in the indices and indices overview resources.

Until we find a better solution, hard code the index suffix for restored archive indices to make it possible to search in restored indices and close/delete them.

Fixes #2058
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment