Meta-issue: StatefulSet refactoring

[sebgl]

The work on refactoring our orchestration layer to use StatefulSets started with #1173, in the statefulset-refactoring feature branch.

Steps:

• Migrate from pods to StatefulSets
• Support for StatefulSet scale up/down
• Support StatefulSets rolling upgrades (PV reuse)
• Improve zen1 & zen2 support
• Refactor/cleanup driver & pod creation logic towards simplification
• Get E2E tests green
• Merge the statefulset-refactoring feature branch into master

Then, several tasks can be achieved in parallel:

• Add missing StatefulSets unit tests
• Improve zen1 support
• Remove PVCs when removing StatefulSets
• Improve zen1/zen2 support in rolling upgrades
• Make sure emptyDirs are correctly supported
• Account for maxSurge during StatefulSet replacements
• Add more rolling upgrade e2e tests
• Warn user about impossible PVC resize
• Improve rolling upgrade health checks
• Rely on maxUnavailable during rolling upgrades
• Implement grow and shrink - delayed to later/non-urgent

[sebgl]

I'm closing this now. Most issues have been solved already. #1292 is in progress and will be solved pretty soon.

The only remaining issue is #1290, to support cluster mutations in a grow-and-shrink fashion (instead of rolling upgrades). This is partially implemented when adding/removing StatefulSets, but not when replacing Pods of an existing StatefulSet, for which we always default to rolling upgrades. We may decide to not implement it though, since rolling upgrades is the default documented way of operating Elasticsearch.