Issue/2066 Elasticsearch / Elastis4s Upgrade

[t83714]

What this PR does

Fixes #2066

Upgrade Elasticsearch to v6.5.x.

• This will help us to fix issue Elasticsearch Simple String Query can't handle stopwords probably when defaultOperator is AND #2064
• Current version is old & has a vulnerability (Upgrade elasticsearch/elastic4s version #2024 )
• Also fixed Re-add ability for elastic search to pull backups #1571 & Use standalone Elasticsearch instances for Unit Tests #1454

Todos:

• Upgrade Elasticsearch to v6.5.1 v6.5.4
• Upgrade Elastic4s to v6.5.1
• Make code changes to suit new version
  • We should try use main release Elastic4s rather than our forked version --- unless there are any blocking bugs
• Make sure all test cases are running ok
  • If really hard to get embeded node to work with new version, look at possible Docker based test solution (seems used by Elastic4s)

Changes Summary:

• Upgrade Elasticsearch to v6.5.4
  • Removed x-pack
  • Set auto_create_index to false (no exception)
• Upgrade Elastic4s to v6.5.1
  • Dumped my forked version & now use main release Elastic4s
• Re-wrote indexer code to work with new versions
  • It seems es v6.5 will sometimes set index to closed after recovery. It won't happen very often but you will see if keep restarting the pod. Because of this, indexer now will try to re-open index if it's closed on startup.
  • Fixes issues with snapshots: indexer will checking whether an index is ready after restore. It prevent indexer from exiting on checking index data immediately after snapshot restore
• Re-wrote search API code to work with new versions
• Removed local node for testing --- now use Docker container
• Helm Chart & Docker images adjustment for using offical elasticsearch docker image

Regarding getting rid of magda-elastic-search image:

• Possible --- Offical image is used in CI for testing as tests happens before the build
  • We could probably insert files (e.g. synonym files) to pod via configMap. However, k8s has a 1 mb size limitation of object storage (Size limit for ConfigMap kubernetes/kubernetes#19781). We probably could load files remotely --- but it might not be the desired solution.

Test

Run sbt tests locally

• cd magda-int-test
• docker-compose up -d
• yarn test

Finally Test Site is Ready:

https://issue-2066.dev.magda.io

Checklist

• There are unit tests to verify my changes are correct
• I've updated CHANGES.md with what I changed.
• I've linked this PR to an issue in ZenHub (core dev team only)