New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JWT auth configuration in 6.2.4 hangs Elasticsearch #403

Closed
dominic-lear opened this Issue Jan 5, 2019 · 4 comments

Comments

Projects
None yet
3 participants
@dominic-lear
Copy link

dominic-lear commented Jan 5, 2019

Hi, im trying to set up JWT auth with elasticsearch with the below details but when starting elasticsearch it hangs at Settings observer refreshing. I have this config working on another computer but that might be with slightly different versions or ROR/java. Can you provide any insight into what the issue may be?


Java version

java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

Elasticsearch version: 6.2.4
Readonlyrest version: https://readonlyrest-data.s3.amazonaws.com/build/1.16.32/readonlyrest-1.16.32_es6.2.4.zip

Readonlyrest config

readonlyrest:
  access_control_rules:
    - name: "BLock 1 - Allow"
      type: allow
      hosts: ["127.0.0.1", "192.168.10.1", "192.168.10.190"] 
      jwt_auth:
        name: "jwt_provider_1"
        roles: ["user"]
    - name: "Block 2"
      type: forbid
  jwt:
    - name: jwt_provider_1
      signature_algo: HMAC # can be NONE, RSA, HMAC (default), and EC
      signature_key: "my-key"
      user_claim: user
      roles_claim: roles # JSON-path style
      header_name: Authorization
  ssl:
    keystore_file: "keystore.jks"
    keystore_pass: readonlyrest
    key_pass: readonlyrest

When starting Elasticsearch it hangs at

2019-01-05T21:29:59,570][INFO ][o.e.n.Node               ] [rate-search-local-1] initializing ...
[2019-01-05T21:29:59,693][INFO ][o.e.e.NodeEnvironment    ] [rate-search-local-1] using [1] data paths, mounts [[/ (/dev/mapper/vagrant--vg-root)]], net usable_space [51.1gb], net total_space [61.7gb], types [ext4]
[2019-01-05T21:29:59,704][INFO ][o.e.e.NodeEnvironment    ] [rate-search-local-1] heap size [247.6mb], compressed ordinary object pointers [true]
[2019-01-05T21:29:59,724][INFO ][o.e.n.Node               ] [rate-search-local-1] node name [rate-search-local-1], node ID [PDJznGXUTuSjmqdLDso3Yg]
[2019-01-05T21:29:59,726][INFO ][o.e.n.Node               ] [rate-search-local-1] version[6.2.4], pid[5601], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Linux/4.15.0-32-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-01-05T21:29:59,728][INFO ][o.e.n.Node               ] [rate-search-local-1] JVM arguments [-Xms256m, -Xmx256m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.enforce.bootstrap.checks=true, -Des.path.home=/home/vagrant/elasticsearch, -Des.path.conf=/home/vagrant/elasticsearch/config]
[2019-01-05T21:30:04,429][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [aggs-matrix-stats]
[2019-01-05T21:30:04,430][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [analysis-common]
[2019-01-05T21:30:04,431][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [ingest-common]
[2019-01-05T21:30:04,433][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [lang-expression]
[2019-01-05T21:30:04,434][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [lang-mustache]
[2019-01-05T21:30:04,435][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [lang-painless]
[2019-01-05T21:30:04,437][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [mapper-extras]
[2019-01-05T21:30:04,437][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [parent-join]
[2019-01-05T21:30:04,438][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [percolator]
[2019-01-05T21:30:04,439][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [rank-eval]
[2019-01-05T21:30:04,440][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [reindex]
[2019-01-05T21:30:04,441][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [repository-url]
[2019-01-05T21:30:04,441][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [transport-netty4]
[2019-01-05T21:30:04,441][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded module [tribe]
[2019-01-05T21:30:04,442][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [readonlyrest]
[2019-01-05T21:30:04,447][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-core]
[2019-01-05T21:30:04,448][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-deprecation]
[2019-01-05T21:30:04,450][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-graph]
[2019-01-05T21:30:04,450][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-logstash]
[2019-01-05T21:30:04,450][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-ml]
[2019-01-05T21:30:04,451][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-monitoring]
[2019-01-05T21:30:04,451][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-security]
[2019-01-05T21:30:04,452][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-upgrade]
[2019-01-05T21:30:04,453][INFO ][o.e.p.PluginsService     ] [rate-search-local-1] loaded plugin [x-pack-watcher]
[2019-01-05T21:30:11,111][INFO ][t.b.r.e.IndexLevelActionFilter] [rate-search-local-1] Settings observer refreshing...

it never gets past Settings observer refreshing

if i remove the lines below, elasticsearch does start correctly.

jwt_auth:
        name: "jwt_provider_1"
        roles: ["user"]
@jbiel

This comment has been minimized.

Copy link

jbiel commented Jan 7, 2019

Thanks Dominic. As you've noted, I reported the same issue in the forums.

@maghibus

This comment has been minimized.

Copy link

maghibus commented Jan 28, 2019

HI,
I think the problem lies elsewhere:
https://stackoverflow.com/questions/137212/how-to-solve-slow-java-securerandom

Instead, running on windows works properly.

@maghibus

This comment has been minimized.

Copy link

maghibus commented Jan 29, 2019

Add -Djava.security.egd=file:/dev/./urandoms in /etc/elasticsearch/jvm.options

@dominic-lear

This comment has been minimized.

Copy link
Author

dominic-lear commented Jan 29, 2019

This does resolve the issue.

[2019-01-29T14:05:30,685][INFO ][t.b.r.e.IndexLevelActionFilter] [rate-search-local-1] Settings observer refreshing...
[2019-01-29T14:06:29,968][INFO ][t.b.r.a.ACL              ] ADDING BLOCK:	{ name: 'BLock 1 - Allow', policy: ALLOW, rules: [hosts, jwt_auth]}

It does take around a minute to boot ES but i think thats an acceptable trade off.

ping @jbiel see the fix above.

Thanks for all the help @maghibus

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment