Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reset locale to C in bin/elasticsearch #6047

Merged
merged 1 commit into from May 27, 2014
Merged

Reset locale to C in bin/elasticsearch #6047

merged 1 commit into from May 27, 2014

Conversation

spinscale
Copy link
Contributor

Because the NetworkExceptionHelper class relies on the english language in
order to extract information and decide whether a certain exception is a
network problem, we need to set the english locale on startup in order
to prevent other locales to circumvent this check.

@jpountz
Copy link
Contributor

jpountz commented May 9, 2014

I'm surprised the exception messages are localized, I thought it was only the case for the getLocalizedMessage method (which is used in toString()), not getMessage()?

@spinscale
Copy link
Contributor Author

Hey,

thats true. I dont know why it happens, but I know it happens, see #5957

Maybe some user/library code does this by accident?

@jpountz
Copy link
Contributor

jpountz commented May 9, 2014

I suspect this error message is generated at the OS level and is just reused by the JVM so I think your PR would fix it!

@jpountz jpountz added v1.1.2 and removed v1.1.2 labels May 13, 2014
@dakrone
Copy link
Member

dakrone commented May 13, 2014

This is the only export statement in our startup files, should we do:

LANG=C
LC_ALL=C

instead to follow the pattern from bin/elasticsearch.in.sh? (ie, no export, just set it locally)

@s1monw
Copy link
Contributor

s1monw commented May 15, 2014

@spinscale what's the status of this?

@s1monw s1monw removed the review label May 15, 2014
@spinscale
Copy link
Contributor Author

@s1monw I just need to test, if Lee's suggestion works flawless under ubuntu (as they use dash and not bash for /bin/sh) and then its ready to push from my side

@spinscale
Copy link
Contributor Author

removed the export and added a bit more explanation

@dakrone
Copy link
Member

dakrone commented May 16, 2014

Hardcoding the locale to C causes indices with non-ascii names to have some issues, for the index "test-weird-index-中文", I get this exception:

[2014-05-16 10:42:07,288][WARN ][cluster.action.shard     ] [Quentin Beck] [test-weird-index-??][0] sending failed shard for [test-weird-index-??][0], node[R7pUZLpKQWmwfjq65z-M-A], [P], s[INITIALIZING], indexUUID [LY4SJInhT3CqBeMZuscziw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][0] failed recovery]; nested: InvalidPathException[Malformed input or input contains unmappable chacraters: /Users/hinmanm/src/elasticsearch/tmp/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0/indices/test-weird-index-??/0/index/write.lock]; ]]
[2014-05-16 10:42:07,288][WARN ][cluster.action.shard     ] [Quentin Beck] [test-weird-index-??][0] received shard failed for [test-weird-index-??][0], node[R7pUZLpKQWmwfjq65z-M-A], [P], s[INITIALIZING], indexUUID [LY4SJInhT3CqBeMZuscziw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[test-weird-index-??][0] failed recovery]; nested: InvalidPathException[Malformed input or input contains unmappable chacraters: /Users/hinmanm/src/elasticsearch/tmp/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0/indices/test-weird-index-??/0/index/write.lock]; ]]
[2014-05-16 10:42:07,291][WARN ][indices.cluster          ] [Quentin Beck] [test-weird-index-??][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [test-weird-index-??][1] failed recovery
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: /Users/hinmanm/src/elasticsearch/tmp/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0/indices/test-weird-index-??/1/index/write.lock
        at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
        at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
        at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
        at java.io.File.toPath(File.java:2186)
        at org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:145)
        at org.apache.lucene.store.Lock.obtain(Lock.java:77)
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:710)
        at org.elasticsearch.index.engine.internal.InternalEngine.createWriter(InternalEngine.java:1279)
        at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:259)
        at org.elasticsearch.index.shard.service.InternalIndexShard.postRecovery(InternalIndexShard.java:687)
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:161)
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
        ... 3 more

And then a ton of these errors repeating. To reproduce, package up this branch, run bin/elasticsearch, then from a checkout of Elasticsearch: LANG=C LC_ALL=C ES_WAIT_ON_MAPPING_CHANGE=true mvn -Dtests.jvms=2 -Dtests.class=\*ElasticsearchRestTests -Dtests.cluster=localhost:9300 clean compile test

@dakrone
Copy link
Member

dakrone commented May 16, 2014

You should also be able to see it by running ES from this branch and doing:

» curl -XPOST 'localhost:9200/test-weird-index-中文/doc/1' -d'{"foo": "bar"}'
{"_index":"test-weird-index-￤ﾸﾭ￦ﾖﾇ","_type":"doc","_id":"1","_version":1,"created":true}%                                                                                                

» indices
health index                   pri rep docs.count docs.deleted store.size pri.store.size
yellow test-weird-index-￤ﾸﾭ￦ﾖﾇ   5   1          1            0      2.6kb          2.6kb

And in the logs:

[2014-05-16 11:07:08,699][INFO ][cluster.metadata         ] [Anomaly] [test-weird-index-??????] creating index, cause [auto(index api)], shards [5]/[1], mappings []
[2014-05-16 11:07:09,184][INFO ][cluster.metadata         ] [Anomaly] [test-weird-index-??????] update_mapping [doc] (dynamic)

@kevinkluge kevinkluge added v1.3.0 and removed v1.2.0 labels May 16, 2014
@spinscale
Copy link
Contributor Author

as suggested by robert, I set the locale to en_EN.UTF-8 - @dakrone maybe you can rerun your tests?

@dakrone
Copy link
Member

dakrone commented May 19, 2014

Tests are much happier now, +1

@s1monw
Copy link
Contributor

s1monw commented May 19, 2014

LGTM

@spinscale
Startup: Reset locale to en_US.UTF-8 in bin/elasticsearch
5fdb35f 
Because the NetworkExceptionHelper class relies on the english language in
order to extract information and decide whether a certain exception is a
network problem, we need to set the english locale on startup in order
to prevent other locales to circumvent this check.
@spinscale spinscale merged commit 5fdb35f into elastic:master May 27, 2014
@spinscale spinscale changed the title Reset locale to C in bin/elasticsearch Startup: Reset locale to C in bin/elasticsearch May 28, 2014
@marevol marevol mentioned this pull request May 28, 2014
Closed
@dakrone dakrone mentioned this pull request Nov 19, 2014
Closed
@clintongormley clintongormley added the :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts label Jun 7, 2015
@clintongormley clintongormley changed the title Startup: Reset locale to C in bin/elasticsearch Reset locale to C in bin/elasticsearch Jun 7, 2015
@mark-vieira mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@spinscale spinscale

Labels
:Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts >enhancement Team:Delivery Meta label for Delivery team v1.3.0 v2.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@spinscale @jpountz @dakrone @s1monw @clintongormley @kevinkluge @mark-vieira