[CI] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory warnings emitted

[dliappis]

Example:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+bwc-tests/221/console

on an immutable worker (elasticsearch-ci-immutable-ubuntu-1604-1554147007999891542)

we got

07:21:18 Suite: org.elasticsearch.upgrades.CcrRollingUpgradeIT
07:21:18   1> [2019-04-02T04:21:08,986][WARN ][o.e.b.JNANatives         ] [[SUITE-CcrRollingUpgradeIT-seed#[5D7F4DF8F9F6B40A]]] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
07:21:18   1> [2019-04-02T04:21:08,990][WARN ][o.e.b.JNANatives         ] [[SUITE-CcrRollingUpgradeIT-seed#[5D7F4DF8F9F6B40A]]] This can result in part of the JVM being swapped out.
07:21:18   1> [2019-04-02T04:21:08,990][WARN ][o.e.b.JNANatives         ] [[SUITE-CcrRollingUpgradeIT-seed#[5D7F4DF8F9F6B40A]]] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
07:21:18   1> [2019-04-02T04:21:08,991][WARN ][o.e.b.JNANatives         ] [[SUITE-CcrRollingUpgradeIT-seed#[5D7F4DF8F9F6B40A]]] These can be adjusted by modifying /etc/security/limits.conf, for example: 
07:21:18   1> 	# allow user 'jenkins' mlockall
07:21:18   1> 	jenkins soft memlock unlimited
07:21:18   1> 	jenkins hard memlock unlimited

Reproduction command:

./gradlew :x-pack:qa:rolling-upgrade-multi-cluster:v6.7.0#follower#clusterTestRunner -Dtests.seed=5D7F4DF8F9F6B40A -Dtests.class=org.elasticsearch.upgrades.CcrRollingUpgradeIT -Dtests.method="testBiDirectionalIndexFollowing" -Dtests.security.manager=true -Dtests.locale=mk -Dtests.timezone=Australia/Perth -Dcompiler.java=12 -Druntime.java=8

I can reproduce this on a Linux and macOS workstation.

[elasticmachine]

Pinging @elastic/es-core-infra

[droberts195]

It's exactly the same test as #40677, although a different error.

It might be worth checking if #40681 will fix this as well.

[dliappis]

Thanks for the feedback @droberts195. I am confident that Unable to lock JVM Memory: error=12, reason=Cannot allocate memory warning is unrelated to the actual test failure tracked in #40677, so I will relabel this to focus on the warning message (and remain as core/infra/build label).

[martijnvg]

@dliappis This looks like the failure that #40681 addressed. So I think that this issue can be closed.

[dliappis]

@martijnvg Yes I've chatted with @atorok earlier to see if it makes sense to do anything about the WARN for being unable to lock the JVM Memory (which is unrelated to the root cause of the test failure) and have renamed this issue accordingly; @atorok as the warn is related to CI infra, does it make sense to still keep this issue open (and here)?

[alpar-t]

I' leaning towards doing nothing for this. It would be nice not to have these warnings in CI as they can be misleading and lead to lost time, but on the other hand, the configuration we have now is the more common one for someone downloading, building and testing elasticsearch so we should rather have the more common setup in CI

[jasontedor]

Where is the bootstrap.memory_lock setting coming from? This is not enabled by default, and is not our recommended configuration, so I would not expect this to be the more common case.

[jasontedor]

I found the source, I will open a change.

[jasontedor]

I opened #40730.

[alpar-t]

I meant not having memlock unlimited is the more common case when running the build. Sorry for the confusion.