EPIC helm chart self-hosted deployment support

[tiina303]

Tracking so we would know what users are struggling with. Please feel free to update/add

Install failed

• cluster too small - we have cluster requirements now on all pages ( https://posthogusers.slack.com/archives/C01GLBKHKQT/p1632033205050900 (too small cluster) & aws EKS fail #59 (too small cluster))
• https://posthogusers.slack.com/archives/G01H7LGR7D0/p1629390081015800 (postgres helm repo, aws changing their docs underneath us)
• https://posthogusers.slack.com/archives/CT7HXDEG3/p1631604962126200 (k8s 1.22 support, which is DONE now)
• Reinstall (https://posthogusers.slack.com/archives/C01FKPGG5U6/p1631774038046900 (reinstall needs upgrade oddness & https://posthogusers.slack.com/archives/CT7HXDEG3/p1633603105399900 (2nd attempt, chi-clickhouse not coming up - upgrade fixes)

uninstall

• https://posthogusers.slack.com/archives/C02E48PCE0K/p1631625238004000 (namespace deletion finalizers) => Document namespace deletion problem #172

Upgrade failed

• DO / helm: Can't Upgrade posthog#6025 (migrate job to delete + re-run upgrade)

TLS setup issues:

• https://posthogusers.slack.com/archives/CT7HXDEG3/p1631975228176900 (confusion around what hostname should be in aws)
• https://posthogusers.slack.com/archives/C01FKPGG5U6/p1631601029037600 (404s: 2 separate; maybe not tls for 2nd)
• https://posthogusers.slack.com/archives/G01JXEDAL22/p1632726015182800 (???)
• HTTPS certificate error #133

Something broke in prod:

• https://posthog.slack.com/archives/C021HLFPAA2/p1631220219019900 (plug-in server couldn't connect to Redis; restart helped) => /health endpoint issue

Kafka:

• https://posthogusers.slack.com/archives/CT7HXDEG3/p1632561949288200 (kafka out of space & resizing didn't work as expected) & https://posthogusers.slack.com/archives/C01FKPGG5U6/p1632577233112700 + issues in comments => we have kafka runbook docs now + changing default sizes atm

Other:

• dry-run crds missing failure - expected no matches for kind "ClickHouseInstallation" in version "clickhouse.altinity.com/v1" #91
• Lookup usage / increase disk size: https://posthogusers.slack.com/archives/CT7HXDEG3/p1632290353227900

[macobo]

Kafka resizing/configuration is a p0 thing IMO - seems like a lot of installations run into this, including ourselves

Slack threads:

• https://posthogusers.slack.com/archives/G01JXEDAL22/p1632708757181700
• https://posthogusers.slack.com/archives/CT7HXDEG3/p1632336279240900
• https://posthogusers.slack.com/archives/C026SNNE430/p1628192801014300
• https://github.com/PostHog/product-internal/issues/54

[macobo]

Another slack issue: https://app.slack.com/client/TT68H7WHY/CT7HXDEG3/thread/CT7HXDEG3-1632993135.320100

[tiina303]

Updated the description (recording follow-ups/done), now just the TLS problems that have been a recurring theme in support questions & @guidoiaquinti has been looking into cert manager already on the chart side