Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ES 2.1 - Failed to delete temp translog file #15021

Closed
skearns64 opened this issue Nov 25, 2015 · 1 comment
Closed

ES 2.1 - Failed to delete temp translog file #15021

skearns64 opened this issue Nov 25, 2015 · 1 comment

Comments

@skearns64
Copy link
Contributor

After upgrading to 2.1, when starting my 1-node dev cluster, I often see the error below from each index that had recent writes before the previous graceful (ctrl-c) shutdown. It seems that these files really do not exist.

I have Marvel installed, but the issue occurs for non-marvel indexes as well. Here's the log from my most recent start.

[2015-11-25 10:09:00,073][INFO ][node                     ] [Cyclone] version[2.1.0], pid[21953], build[72cd1f1/2015-11-18T22:40:03Z]
[2015-11-25 10:09:00,074][INFO ][node                     ] [Cyclone] initializing ...
[2015-11-25 10:09:00,416][INFO ][plugins                  ] [Cyclone] loaded [marvel-agent, license], sites []
[2015-11-25 10:09:00,443][INFO ][env                      ] [Cyclone] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [81.9gb], net total_space [232.6gb], spins? [unknown], types [hfs]
[2015-11-25 10:09:03,358][INFO ][node                     ] [Cyclone] initialized
[2015-11-25 10:09:03,358][INFO ][node                     ] [Cyclone] starting ...
[2015-11-25 10:09:03,687][INFO ][transport                ] [Cyclone] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[fe80::1]:9300}, {[::1]:9300}
[2015-11-25 10:09:03,701][INFO ][discovery                ] [Cyclone] elasticsearch/FEmp7CE8QNuHT_0JujYmrQ
[2015-11-25 10:09:06,743][INFO ][cluster.service          ] [Cyclone] new_master {Cyclone}{FEmp7CE8QNuHT_0JujYmrQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2015-11-25 10:09:06,763][INFO ][http                     ] [Cyclone] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[fe80::1]:9200}, {[::1]:9200}
[2015-11-25 10:09:06,764][INFO ][node                     ] [Cyclone] started
[2015-11-25 10:09:07,088][INFO ][license.plugin.core      ] [Cyclone] license [cf52acc6-1817-4e1c-b97b-0170927a54b2] - valid
[2015-11-25 10:09:07,094][ERROR][license.plugin.core      ] [Cyclone] 
#
# License will expire on [Friday, December 25, 2015]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
# 
# Commercial plugins operate with reduced functionality on license expiration:
# - marvel
#  - The agent will stop collecting cluster and indices metrics
[2015-11-25 10:09:07,169][INFO ][gateway                  ] [Cyclone] recovered [3] indices into cluster_state
[2015-11-25 10:09:07,827][WARN ][index.translog           ] [Cyclone] [.marvel-es-data][0] failed to delete temp file /skearns/es/elasticsearch-2.1.0/data/elasticsearch/nodes/0/indices/.marvel-es-data/0/translog/translog-5891737687262078999.tlog
java.nio.file.NoSuchFileException: /skearns/es/elasticsearch-2.1.0/data/elasticsearch/nodes/0/indices/.marvel-es-data/0/translog/translog-5891737687262078999.tlog
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1079)
    at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:324)
    at org.elasticsearch.index.translog.Translog.<init>(Translog.java:166)
    at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:209)
    at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
    at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
    at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1408)
    at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1403)
    at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:906)
    at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:883)
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:245)
    at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
    at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[2015-11-25 10:09:07,838][WARN ][index.translog           ] [Cyclone] [.kibana][0] failed to delete temp file /skearns/es/elasticsearch-2.1.0/data/elasticsearch/nodes/0/indices/.kibana/0/translog/translog-222310247107863220.tlog
java.nio.file.NoSuchFileException: /skearns/es/elasticsearch-2.1.0/data/elasticsearch/nodes/0/indices/.kibana/0/translog/translog-222310247107863220.tlog
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1079)
    at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:324)
    at org.elasticsearch.index.translog.Translog.<init>(Translog.java:166)
    at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:209)
    at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
    at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
    at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1408)
    at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1403)
    at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:906)
    at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:883)
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:245)
    at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
    at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[2015-11-25 10:09:07,919][WARN ][index.translog           ] [Cyclone] [.marvel-es-2015.11.25][0] failed to delete temp file /skearns/es/elasticsearch-2.1.0/data/elasticsearch/nodes/0/indices/.marvel-es-2015.11.25/0/translog/translog-6094222780037808919.tlog
java.nio.file.NoSuchFileException: /skearns/es/elasticsearch-2.1.0/data/elasticsearch/nodes/0/indices/.marvel-es-2015.11.25/0/translog/translog-6094222780037808919.tlog
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
    at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
    at java.nio.file.Files.delete(Files.java:1079)
    at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:324)
    at org.elasticsearch.index.translog.Translog.<init>(Translog.java:166)
    at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:209)
    at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:152)
    at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
    at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1408)
    at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1403)
    at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:906)
    at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:883)
    at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:245)
    at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
    at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants