Biocache test infrastructure

[ansell]

The current biocache test infrastructure is only setup for a single server for ala-hub and a single server for biocache-service and a load balancer. It relies on access to the production databases/indexes. This makes it impossible to test any changes to the other components, such as Zookeeper, Solr, Cassandra, Biocache-store and Jenkins.

Issues for this project include:

• Create biocache-service test instance. Ansible definition in ansible-inventories as test-bws-b5. Also gave biocache-service instance access to auth.ala.org.au in authorised systems. This currently points at the production cassandra/solr/zookeeper due to lack of budget for setting up those parts of the biocache test infrastructure.
• Create ala-hub test instance. Ansible definition in ansible-inventories as test-bhub-b5. Also gave ala-hub instance access to auth.ala.org.au using authorised systems.
• Setup biocache-test.ala.org.au/biocache-ws-test.ala.org.au shared load balancer. biocache-test.ala.org.au and biocache-ws-test.ala.org.au point to the same load balancer, with two separate target groups for /ws and /. Unlike production biocache-which has multiple targets, biocache-ws-test queries all go to a single server, so downloads are not sent to a different dedicated server as in production. This would need to be changed to match production if stress tests were part of the accepted requirements for the biocache test infrastructure.
• Create solr-cloud-test and cassandra-cloud-test AWS security groups https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/55
• Manage AWS security group configurations and allocations in ansible https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/6
• Add zookeeper configuration tasks to ansible role Add zookeeper configuration tasks to ansible role ala-install#197
• Setup zookeeper test cluster https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/31
• Setup cassandra test cluster https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/30
• Setup solr test cluster https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/32
• Setup biocache store test instances https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/56
• biocache-properties creates directories but doesn't change their owner biocache-properties creates directories but doesn't change their owner ala-install#172
• Setup biocache test jenkins https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/54
• Create Jenkins job to push data from production cassandra to test cassandra https://github.com/AtlasOfLivingAustralia/ala-infrastructure/issues/33
• Add Cassandra-3 source to cmigrate Add Cassandra-3 source cmigrate#1

Data load related issues are currently being debugged in production in the absence of this infrastructure being available.

[ansell]

This is stalled due to lack of funds in the budget for test systems:

https://github.com/AtlasOfLivingAustralia/ALA-Work-Tracking/issues/176