Fix out of memory #775
Fix out of memory #775
Conversation
Travis uses openvz virtualization, and therefore we can't modify kernel settings See: elastic/elasticsearch#4978 See: http://changelog.travis-ci.com/post/45177235333/builds-now-running-on-openvz
bootstrap.mlockall is used, no reason to use ulimit See: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html
Don't write test data to hdd See: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-store.html#store-memory
Actually two is really enough. One of them shuts down in testNodeShutdown and second - in testClusterShutdown. If keep using three, OutOfMemory always happens, regardless of used settings. So, it's a lot much simpler to remove one node than try to find wtf is going on in travis.
Don't ever accept connections from same datacenter in test builds.
New script waits while both 9200 and 9201 nodes are started. Then waits 30 seconds and check them again. If some node suddenly unavailable, trying to restart cluster. If cluster was restarted 10 times and still no success, it's fail, go and fix code.
Without this, java will spawn new threads until die See: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html#processors
Limiting stack size allow java to spawn more threads, so this fixes OutOfMemory in travis builds
This allows to be sure that all tests is not depends on each other. Also, prefer use _createIndex instead of manual creation.
Sleep before assert allows to be sure that all nodes in cluster has created type
1 similar comment
|
Nice, thanks. What would be the consequence of still having 3 instances for testing? Would we run into any limitations because of the criterias you mentioned above? |
|
Yep, there will be problems. I was trying a lot not do this, but at the end I failed to get stable 3-nodes cluster. Third node always either dies immediately after start or in the middle of test suite, regardless of used settings. It dies even if phpunit will sleep between tests and run them very slow. Anyway, I'm sure, that run multiple nodes on machine with such low performance it's not an usual way to use elasticsearch, so it's good thing at least two instances can be started. |
|
Ok, good to know. So I assume the issues appear on travis and not on the local machine? |
|
BTW: The only issue I have with the new tests is that they take quite a bit longer. I assume this is because of the removal of the indices. But here stability goes over speed. |
|
Yes, I still can't reproduce it in the virtualbox. I think, it's could be because of virtualization system difference. In travis this is openvz, and in virtualbox this is em... virtualboxvz? I'm not good at this, so I will stop making further assumptions :) |
|
I'm happy it works now ;-) Thx a lot. |
This PR fixes OutOfMemory exception (which indeed must be named like OutOfThreadLimit, because problem is not related to memory at all) in travis builds. Last 132 builds of last commit (22 runs * 6 php versions) were successfull, except 3, that were failed in
provision.shon stepapt-get installorpip installdue to network issues.Summary of changes:
processors: 1settingulimit -lmust be either an integer 256 or "unlimited", but string "256M" was used)vm.max_map_countwas removed, because we can't use sysctl inside travis machine_createIndexnow will generate unique name for created index, if name was not provided as first argumentindex.store.type: memorysettingTest\Base::tearDownnow performs deletion of all indices and caches between tests