New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[migration v6.5] Another Kibana instance appears to be migrating the index #25464

Open
CamiloSierraH opened this Issue Nov 9, 2018 · 15 comments

Comments

Projects
None yet
@CamiloSierraH
Copy link

CamiloSierraH commented Nov 9, 2018

Kibana version: 6.5.0

Elasticsearch version: 6.5.0

Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):
From source

Describe the bug:
I'm using Elasticsearch and Kibana in v6.4.3 and i'm testing to migrate to v6.5.0.
When i start Kibana for the fist time un v6.5.0 i stop the process during the migration and i have an empty browser page for Kibana

Steps to reproduce:
To reproduce, i start kibana and i stop when the logs where on this stage:

  log   [14:00:01.131] [info][migrations] Creating index .kibana_2.
  log   [14:00:01.221] [info][migrations] Reindexing .kibana to .kibana_1

At that point the response of aliases cat request are :

.security .security-6 - - -

and if i try to restart the Kibana service, i have an empty page in the browser and in the logs i have this message:

log   [14:00:20.457] [warning][migrations] Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana.

capture d ecran 2018-11-09 a 10 53 45

I delete .kibana_2 index as mentioned in the logs, using this Curl request:

curl -XDELETE 'http://localhost:9200/.kibana_2'  --header "content-type: application/JSON" -u elastic -p

I restart Kibana and i have this message:

[warning][migrations] Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana.

I delete .kibana_1 index as mentioned in the logs, using this Curl request:

curl -XDELETE 'http://localhost:9200/.kibana_1'  --header "content-type: application/JSON" -u elastic -p

before deleting the index .kibana_1 we need to verify that in my elasticsearch server i have the index .kibana ?
I ask this because if i understand .kibana_1 is the copy of .kibana and .kibana is deleted when the migration is finished. So if i delete as requested .kibana_1 and .kibana was already deleted i may lose all the Dashboards/visualization i have stored? i am right?

I restart Kibana and this time everything works, Kibana is back on the browser, and in the logs i have the logs:

[migration] finished in 688ms.

@bhavyarm
Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

@elasticmachine

This comment has been minimized.

Copy link

elasticmachine commented Nov 14, 2018

@tylersmalley

This comment has been minimized.

Copy link
Member

tylersmalley commented Nov 14, 2018

@CamiloSierraH - thanks for the report. This issue is caused by stopping the process which is in charge of handling the migration. This "locking" is to handle having multiple Kibana instances.

With this issue I see two possible problems:

  • We should add a note to "Reindexing .kibana to .kibana_X" stating not to stop the Kibana process.
  • The index name in the messaging for subsequent re-tries during of the re-index might not be correct.

log [14:00:20.457] [warning][migrations] Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana.

There is more information available here: https://www.elastic.co/guide/en/kibana/current/upgrade-migrations.html

@Hendler

This comment has been minimized.

Copy link

Hendler commented Nov 15, 2018

same issue in a docker/dev environment

@lnx01

This comment has been minimized.

Copy link

lnx01 commented Nov 15, 2018

Same issue - upgraded from 6.4.0 to 6.5.0 using DEB package - appears to be stuck on "Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_2 and restarting Kibana."

Deleting .kibana_2 and restarting causes the same thing to happen, gets stuck on message above.

Kibana UI says "Kibana server is not ready yet" -- cannot access /status either, same message.

@miko-code

This comment has been minimized.

Copy link

miko-code commented Nov 15, 2018

Same issue as @lnx01 upgrade from 6.4.x to 6.5.0

@gmeriaux

This comment has been minimized.

Copy link

gmeriaux commented Nov 15, 2018

I have the same issue and was working on my test instance. Actually, I have no access to kibana.
Have you got a quick solution to find back my UI? Is it possible to downgrade ELK stack or just kibana?

@CamiloSierraH

This comment has been minimized.

Copy link
Author

CamiloSierraH commented Nov 15, 2018

@gmeriaux you need to follow this steps to have back your kibana instance -> https://www.elastic.co/guide/en/kibana/current/release-notes-6.5.0.html#known-issues-6.5.0

@gheffern

This comment has been minimized.

Copy link

gheffern commented Nov 15, 2018

@gmeriaux I had success with just downgrading kibana and removing the indexes .kibana_1 and .kibana_2

@taku333

This comment has been minimized.

Copy link

taku333 commented Nov 20, 2018

@CamiloSierraH,@gheffern
thanks!!!
l am having trouble upgrading in the Windows environment(6.4.3 ⇨6.5.0)

.kibna2 indice delete after started kibna

@taku333

This comment has been minimized.

Copy link

taku333 commented Nov 21, 2018

There. was no. problem with version 6.5.1

I am having success upgrading in Windows environment(6.4.3⇨6.5.1)

@jorgelbg

This comment has been minimized.

Copy link

jorgelbg commented Dec 8, 2018

Found a similar issue while upgrading. Turns out that was related to a .tasks closed index. Kibana was failing with an index_closed_exception this index is not usually used by Kibana (was closed automatically by curator a long time ago).

pevma added a commit to StamusNetworks/SELKS that referenced this issue Dec 14, 2018

@LiaoTung

This comment has been minimized.

Copy link

LiaoTung commented Jan 8, 2019

I noticed that Kibana should be at full stop before deleting the indices. Although Kibana remained slow for a few minutes right before restarting - perhaps to reconstruct the both indices - it came up with all the data intact.

$ curl-XGET "https://localhost:9200/_cat/indices"| grep kibana
...
green open .kibana_2 kVo3hhokTP2hVUSfmPkGVA 1 0 181 0 184.2kb 184.2kb
green open .kibana_1 mHhRaCqKStq6bL1qRLxMVA 1 0 178 0 170.9kb 170.9kb

@walkerxk

This comment has been minimized.

Copy link

walkerxk commented Jan 28, 2019

i have the error message: "Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."
after i use curl -XDELETE http://localhost:9200/.kibana_1 to delete the index, and restart kibana.,i get the same message.
the version of elk are all 6.5.4.
image

@soar

This comment has been minimized.

Copy link

soar commented Feb 5, 2019

I've faced the same problem while upgrading from 6.4.2 to 6.5.4

@Timmy93

This comment has been minimized.

Copy link

Timmy93 commented Feb 5, 2019

I had the same problem migrating from 6.4.3 to 6.6.0.

I solved deleting the 3 indexes: .kibana, .kibana_1 and .kibana_2 and restarting the kibana service.

I used the following command from linux bash:
curl -X DELETE "localhost:9200/.kibana_2" && curl -X DELETE "localhost:9200/.kibana_1" && curl -X DELETE "localhost:9200/.kibana"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment