path.repo not being recognized?

[CoryBond]

So today I wanted to backup our teams kibana index because it was wiped over the weekend and we wanted to make sure we could recover from this if it ever happened again. I went onto the snapshot and restore doc (https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-snapshots.html) on the elasticsearch website for version 1.7 and tried to follow the instructions but have been failing to get a file repo setup.

Specs: elasticsearch 1.7.0, java 8, red hat enterprise linux.

After setting the field ( path.repo: ["opt/velocity/backups"] ) in elasticsearch.yml I do this curl command:

curl -XPUT http://localhost:9200/_snapshot/backups -d '{ "type": "fs", "settings": { "location": "/opt/velocity/backups" } }'

This results in a RemoteTransportException as so:

{"error":"RemoteTransportException[[Ectokid][inet[/169.176.42.123:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[backups] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [backups] location [/opt/velocity/backups] doesn't match any of the locations specified by path.repo because this setting is empty\n at org.elasticsearch.repositories.fs.FsRepository.()\n at org.elasticsearch.repositories.fs.FsRepository\n at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: RepositoryException[[backups] location [/opt/velocity/backups] doesn't match any of the locations specified by path.repo because this setting is empty]; ","status":500}{"error":"RemoteTransportException[[Ectokid][inet[/169.176.42.123:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[backups] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [backups] location [/opt/velocity/backups] doesn't match any of the locations specified by path.repo because this setting is empty\n at org.elasticsearch.repositories.fs.FsRepository.()\n at org.elasticsearch.repositories.fs.FsRepository\n at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: RepositoryException[[backups] location [/opt/velocity/backups] doesn't match any of the locations specified by path.repo because this setting is empty]

It seems as though elasticsearch is not recognizing the path.repo field I setup in elasticsearch.yml. Basically now I am stuck because I believe I have correctly setup my config. I know the path is exists.

I tried playing around with the syntax a little bit like changing location to be "backups" and changing path.repo to be "opt/velocity/backups" (without the bracket) but nothing I have tried works.

I think I read somewhere online that a person was able to set the path.repo field by command line but I don't see any good documentation mentioning this on the elasticsearch docs.

Is there something I am missing here? I hope my issue is a simple config or syntax thing which can be easily fixed.

[imotov]

Did you add the path.repo setting to all data and master nodes and did you restart these nodes after making the change?

[CoryBond]

I only have a local node. I used the shutdown api and then started up elasticsearch. I still see the issue however.

Also thanks for helping me out.

[imotov]

Oh, sorry, I didn't notice when I first looked at the ticket. It should be

path.repo: ["/opt/velocity/backups"]

Please, notice the leading / before opt

[CoryBond]

I have tried adding the leading / but that doesn't work either. I still get the same error.

[imotov]

Can you try one more time adding leading "/" to both elasticsearch.yml and location in the register repository request, restart the node and if you still get an error, please post here the error that you are getting, the log message that is getting generated if any? Please, make sure that there are no spaces before path in elasticsearch.yml

[CoryBond]

Im sorry imotov but I just tried adding "/" to both the location in the curl command and to the elasticsearch.yml. Neither successfully work and it looks like the same message pops up again:

[velo@ny4ufxvel16 bin]$ curl -XPUT http://localhost:9200/_snapshot/backups -d '{ "type": "fs", "settings": { "location": "/opt/velocity/backups" } }'
{"error":"RemoteTransportException[[Ectokid][inet[/169.176.42.123:9300]][cluster:admin/repository/put]]; nested: RepositoryException[[backups] failed to create repository]; nested: CreationException[Guice creation errors:\n\n1) Error injecting constructor, org.elasticsearch.repositories.RepositoryException: [backups] location [/opt/velocity/backups] doesn't match any of the locations specified by path.repo because this setting is empty\n at org.elasticsearch.repositories.fs.FsRepository.()\n at org.elasticsearch.repositories.fs.FsRepository\n at Key[type=org.elasticsearch.repositories.Repository, annotation=[none]]\n\n1 error]; nested: RepositoryException[[backups] location [/opt/velocity/backups] doesn't match any of the locations specified by path.repo because this setting is empty]; ","status":500}

I am looking through the log files after executing the curl command but it seems nothing is being written to them. ls -lrt shows no changes to log sizes and looking at each log file individually shows they were only modified a few minutes before executing the curl command.

I am even tried putting path.repo at the top of the elasticsearch.yml file to see if that helps. I know the elasticsearch.yml is being loaded because one of the configurations in it (script.groovy.sandbox.enabled: true) is allowing us to use scripted fields in kibana.

[imotov]

OK, it looks like the setting is not getting picked up from the elaticsearch.yml. Could you run curl "localhost:9200/_nodes/settings?pretty" to see where paths are pointing to make sure you are changing the right config file and could you send this config file to me? I tested it with a fresh installation of 1.7.0 and everything seems to work fine.

[CoryBond]

First, let me post what I got back:

[velo@ny4ufxvel16 config]$ curl "localhost:9200/_nodes/settings?pretty"
{
"cluster_name" : "elasticsearch",
"nodes" : {
"Ill8ykIfQGacAvtZr43flA" : {
"name" : "logstash-ny4ufxvel16.dev.sti-24805-13246",
"transport_address" : "inet[/10.99.92.147:9301]",
"host" : "ny4ufxvel16.dev.sti",
"ip" : "10.99.92.147",
"version" : "1.5.1",
"build" : "5e38401",
"attributes" : {
"data" : "false",
"client" : "true"
},
"settings" : {
"node" : {
"name" : "logstash-ny4ufxvel16.dev.sti-24805-13246",
"client" : "true"
},
"cluster" : {
"name" : "elasticsearch"
},
"path" : {
"logs" : "/opt/velocity/logstash-1.5.1/bin/logs"
},
"discovery" : {
"zen" : {
"ping" : {
"multicast" : {
"enabled" : "false"
},
"unicast" : {
"hosts" : "localhost:9300,localhost:9301,localhost:9302,localhost:9303,localhost:9304,localhost:9305"
}
}
}
},
"name" : "logstash-ny4ufxvel16.dev.sti-24805-13246",
"http" : {
"enabled" : "false"
},
"client" : {
"type" : "node",
"transport" : {
"sniff" : "false"
}
}
}
},
"YGrmZ3TgQG2JWjsjq1CTdA" : {
"name" : "Ectokid",
"transport_address" : "inet[/169.176.42.123:9300]",
"host" : "ny4dlsor01.dev.sti",
"ip" : "169.176.42.123",
"version" : "1.6.0",
"build" : "cdd3ac4",
"http_address" : "inet[/169.176.42.123:9200]",
"settings" : {
"name" : "Ectokid",
"path" : {
"logs" : "/export/tb76300/elk/elasticsearch-1.6.0/logs",
"home" : "/export/tb76300/elk/elasticsearch-1.6.0"
},
"cluster" : {
"name" : "elasticsearch"
},
"client" : {
"type" : "node"
},
"foreground" : "yes",
"config" : {
"ignore_system_properties" : "true"
}
}
},
"sBNtWMIrRbSyHEB9WHcmUA" : {
"name" : "Velocity Server Logs",
"transport_address" : "inet[/10.99.92.147:9300]",
"host" : "ny4ufxvel16.dev.sti",
"ip" : "10.99.92.147",
"version" : "1.7.0",
"build" : "929b973",
"http_address" : "inet[/10.99.92.147:9200]",
"settings" : {
"path" : {
"logs" : "/opt/velocity/elasticsearch/logs",
"home" : "/opt/velocity/elasticsearch",
"repo" : [ "/opt/velocity/backups" ]
},
"node" : {
"name" : "Velocity Server Logs",
"data" : "true"
},
"cluster" : {
"name" : "elasticsearch"
},
"name" : "Velocity Server Logs",
"http" : {
"cors" : {
"allow-origin" : "/.*/",
"enabled" : "true"
}
},
"index" : {
"number_of_shards" : "5",
"number_of_replicas" : "1"
},
"client" : {
"type" : "node"
},
"foreground" : "yes",
"config" : {
"ignore_system_properties" : "true"
},
"script" : {
"groovy" : {
"sandbox" : {
"enabled" : "true"
}
}
}
}
},
"AOaEZ9uHSJu9iI1WxQn2jw" : {
"name" : "logstash-ny4dlsor01.dev.sti-5862-13484",
"transport_address" : "inet[/169.176.42.123:9301]",
"host" : "ny4dlsor01.dev.sti",
"ip" : "169.176.42.123",
"version" : "1.5.1",
"build" : "5e38401",
"attributes" : {
"client" : "true",
"data" : "false"
},
"settings" : {
"node" : {
"name" : "logstash-ny4dlsor01.dev.sti-5862-13484",
"client" : "true"
},
"cluster" : {
"name" : "elasticsearch"
},
"path" : {
"logs" : "/export/tb76300/elk/logs"
},
"discovery" : {
"zen" : {
"ping" : {
"multicast" : {
"enabled" : "false"
},
"unicast" : {
"hosts" : "localhost:9300,localhost:9301,localhost:9302,localhost:9303,localhost:9304,localhost:9305"
}
}
}
},
"name" : "logstash-ny4dlsor01.dev.sti-5862-13484",
"http" : {
"enabled" : "false"
},
"client" : {
"type" : "node",
"transport" : {
"sniff" : "false"
}
}
}
}
}
}

[CoryBond]

I see this in the file "repo" : [ "/opt/velocity/backups" ]

[CoryBond]

When I try to perform this curl command though I get this:

[velo@ny4ufxvel16 config]$ curl -XPUT localhost:9200/_snapshot/backups/snapshot_1 -d '{ "indices": ".kibana", "include_global_state": false}'
{"error":"RemoteTransportException[[Ectokid][inet[/169.176.42.123:9300]][cluster:admin/snapshot/create]]; nested: RepositoryMissingException[[backups] missing]; ","status":404}

It claims the repo doesn't exist...

[imotov]

You need to add this setting to all data and master nodes. It looks like you didn't add it to Ectokid, 169.176.42.123:9200.

[CoryBond]

Im going to sound ignorant by asking but how do I add this setting to Ectokid?... If I may also ask, what is the purpose of the Ectokid node? The only node I familiar in this list is the Velocity Server Logs because that is the one that has my data and is configured in my elasticsearch.yml.

[imotov]

It depends on how you run this server. Is it stand-alone server or it runs embedded somewhere? If it's standalone service, you can modify its config file that can be found in /export/tb76300/elk/elasticsearch-1.6.0/config/elasticsearch.yml. If it's embedded, it really depends on the application it's embedded into.

[CoryBond]

Right so I am a little confused about this. We don't seem to have a directory called /export/ anywhere in our file-system. I tried "find . -type d -name "export" -ls" and all I am getting is permission denied for some directories.

Also tried "find . ! -readable -prune -type d -name "elk" -ls" but I don't get anything

[imotov]

Try running ps -aef | grep java on this machine. If it runs embedded elasticsearch it has to run jvm somehow.

[CoryBond]

I also greped over elasticsearch.

[velo@ny4ufxvel16 bin]$ ps -aef | grep java | grep elasticsearch
velo 19456 1 80 11:18 pts/6 00:00:32 /usr/java/1.8.0_31l64/bin/java -Xms24g -Xmx24g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.foreground=yes -Des.path.home=/opt/velocity/elasticsearch -cp :/opt/velocity/elasticsearch/lib/elasticsearch-1.7.0.jar:/opt/velocity/elasticsearch/lib/:/opt/velocity/elasticsearch/lib/sigar/ org.elasticsearch.bootstrap.Elasticsearch

[imotov]

You need to run these commands on 169.176.42.123 where the misconfigured server is located, not on 10.99.92.147, where you already configured it properly.

[CoryBond]

Hey Thanks Imotov I got the snapshot and recovery api to work. I now see that the reason for my problems is because my ES was in using the default multicast mode rather then unicast mode. It was picking up all sorts of nodes across our network that weren't related to my teams work.

I looked over this documentation here: https://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html and see that changing the modes this is a necessary configuration that our own ES needs.

[imotov]

Sounds like I can close this issue then.

[ksripathi]

@imotov

I got same issue but I couldn't resolve it using https://www.elastic.co/guide/en/elasticsearch/guide/current/_important_configuration_changes.html since it not available

Can you suggest what steps needs to be followed

[imotov]

@ksripathi first of all, the best place to ask questions like this is our support forum https://discuss.elastic.co/ We are trying to use github only to track issues and enhancement reports. Creating issues with questions or resurrecting old issues like this creates additional noise that might cause us to miss important bug reports. Saying that, the page that you are referring to is now https://www.elastic.co/guide/en/elasticsearch/guide/master/important-configuration-changes.html . If you have any questions or need additional assistance please don't hesitate to ask on https://discuss.elastic.co/.