Problem with index name handling

[bernd]

Problem description

In 1728772 we introduced a stricter pattern to check if an index is handled by Graylog to avoid touching anything else.

I see two problems with that:

1. The "System/Indices" page still shows other indices. It looks like the pattern is not used when requesting the list of indices.
2. In some cases we actually need to handle indices that do not match the pattern. For example the upcoming archive plugin needs to be able to restore indices with a different name to avoid any name clashes.

Trying to open the details for the graylog_17_restored_archive does not work.

The details do not show up, the spinner keeps spinning.

In the developer tools I see a failed request for the archive because the name does not match the strict pattern.

{
  "type":"ApiError",
  "message":"Index [graylog_17_restored_archive] doesn't look like an index managed by Graylog."
}

Steps to reproduce the problem

1. Use archive plugin in 2.0.0-beta.2 to create an archive for an index.
2. Restore the archive index.
3. Navigate to "System/Indices"
4. See an index named like *_restored_archive
5. Try to click "Show Details / Actions"
6. Spinner does not go away and see errors in the developer console.

Solution Discussion

To avoid showing the non-^graylog_[0-9]+$ indices on the "System/Indices" page, we could just make sure to apply the strict pattern for the index list as well.

The problem is that the archive plugin cannot be used for restores at the moment because of the strict pattern handling. An archived index can be restored but then there will be no index range for it and we also cannot delete it.

One idea would be to make the index name patterns pluggable to allow plugins to inject a new name format.

Any other or better ideas?

Environment

• Graylog Version: 2.0.0-beta.3-SNAPSHOT (882c097)
• Elasticsearch Version: 2.3.0

[kroepke]

I'd be fine with hard coding the archive plugin name format for now and properly fixing this in 2.1.

Personally I consider the index naming non-public API and I'm hesitant to open that up to plugins without a good plan. Especially once we want to allow multiple indices, possibly per stream or other criteria, this will quickly become messy.

How about allowing a suffix of _restored_archive for getting information and recalc of index ranges but disallowing them to be part of the "which is the latest index" logic?

[joschi]

How about allowing a suffix of _restored_archive for getting information and recalc of index ranges but disallowing them to be part of the "which is the latest index" logic?

I'd also prefer that as a quick solution while thinking about a proper solution (maybe also incorporating solutions which do not use Elasticsearch) in a later version.

[bernd]

Thanks for the feedback! I created a PR: #2062