Add st2chatops

[arm4b]

Currently K8s/Helm is missing st2chatops deployment.

Making it Docker and K8s-friendly to read ENV vars outside of st2chatops.env will take some work, (see blocker StackStorm/st2chatops#50).

And so running 1 replica of st2chatops would be doable and nice to have, but having st2chatops in HA is a story for a new dedicated issue and research.

[ghost]

As per documentation, st2chatops can work in HA as a shared component. Trying to the best way to accomplish that, How valid is following a scenario?

1. Deploy native st2chatops container found here to the cluster.
2. Setup environment variable st2chatops to communicate with st2.

Any feedback on the feasibility of this approach and/or whether this can be considered as a viable chatops addition to Stackstorm HA. If we have a positive answer for the above, the next stop is how we can automate this flow using the existing chart to include chatops as well.

[afoninsky]

https://github.com/afoninsky/stackstorm-chatops-playground

I played with implementing chatops into existing stackstorm kubernetes instance with additional telegram adapter.
As I remember everything worked fine. But can't promise it works for now. Could be a good point to start.

[arm4b]

@mosn st2chatops (which relies on hubot) is currently not HA-friendly.

See:

• https://www.reddit.com/r/chatops/comments/2qi0cx/how_to_multiplex_hubot_for_ha/
• https://www.reddit.com/r/chatops/comments/6irq20/how_to_run_hubot_in_ha_highly_available/
• Multiple instances of hubot-slack responding to messages slackapi/hubot-slack#382

What we want to do is just having 1 normally installed instance of st2chatops to have minimal functionality present in K8s deployment.

[arm4b]

As for implementation, ideally to start with defining a Dockerfile for st2chatops here: https://github.com/stackstorm/st2-dockerfiles so we can proceed with deploying it to Docker Hub (https://hub.docker.com/u/stackstorm).

st2chatops configuration could be controlled by passing ENV variables to override defaults from https://github.com/StackStorm/st2chatops/blob/master/st2chatops.env.

Once that's done next step would be creating a K8s resource for st2chatops as well as corresponding Helm values for stackstorm-ha (this repo).

We're definitely very open to contributions and happy if someone can work on that before we'll reach this task ourselves 👍 We'll provide feedback & help along the way.

[ghost]

My approach so far and possibly calls for comments:

• using docker-in-docker, I pull st2chatops to build a new image based on the value provider in values.yaml. This should leave room for further expansion through st2chatops and prevents duplications.

• I decided on taking the standalone approach, rather than combining it with the st2 container. This was one of the pain points I felt first hand after using chatops in production.

My next step:

• I see st2chatops as a component similar to st2web. I assume my next step is to look at how st2web is linked to the components it communicates with and replicates that for st2chatops

Things I still haven't figured out:

• Whether to expose st2chatops through using ingress or having a loadbalancer with the goal of total isolation.

[arm4b]

@mosn
st2chatops is designed to work in standalone mode. That sounds about right if you install only that package without st2. I don't think service/load-balancer is needed for st2chatops as it just connects to ST2 via APIs.

I think docker-in-docker and building st2chatops Docker image every time on startup (if I understand you correctly) sounds like overcomplication and goes against the immutable approach where result can vary depending on build results.

Ideally is to build and deploy produced st2chatops image to some Docker repository and refer to it in K8s deployment. Passing Hubot adapter & settings via ENV variables like it happens https://github.com/StackStorm/st2chatops/blob/master/st2chatops.env should be good enough.

At this point if you want, you can contribute your st2chatops Dockerfile to: https://github.com/stackstorm/st2-dockerfiles so we can review, provide feedback, merge and potentially start building & shipping it to Docker Hub.

[ghost]

@armab Thanks for the info. I will publish my changes as soon as I have it in good shape. I hope the community finds useful.
re. building the image every time, our st2chatop has lots of custom components that we need in our use case. Custom images are needed in our case but definitely not a common one. |

[arm4b]

@mosn Got it.
So in future when we'll have official st2chatops image, you could use FROM stackstorm/st2chatops to build your modified Docker image on top of that + deploy it.
That also means it will be a good idea for us to allow pointing to custom chatops Docker image in Helm values, similar as st2packs:
https://github.com/StackStorm/stackstorm-ha/blob/f7498be460bfc3df50aab75318b1af8f7c05350c/values.yaml#L78-L87

[ghost]

Is there a chatops branch where we can submit progression on this?
There's an example here which is what we tried in our testing. We choose the st2chatops docker for faster integration. You can see the change to the template hte here;

[ghost]

I faced following questions while working on this

• Should chatops component work as as separate container or Integrated inside the StackStorm as we know it? a good example of this can be seen in custom_pack implementation.
• Packaged distributions or ready container: st2chaops package distribution offers an easy to integrate solutions and reduces deployment step but is more error prone to customization in this environment; On the other hand, building from dockers can take time and is not as straight forward but offers better customization.
• values.yaml file: There are a lot of parameters that users need/do not need to watch out for which will impact user experience and ease of use- current values.yaml fine is one of the cleanest I've ever seen. I hope it remains like that post chatops integration :-)

[arm4b]

@mosn Sure, as you already created a Github fork with custom changes, just fill a "Work in Progress" Pull Request against our stackstorm/stackstorm-ha repository and we'll provide feedback/help there.

[ghost]

@armab I'd like to take a few moments to reflect on our current progress.
#55 received good feedback that I can use to proceed with the rest of the template files. I am also testing the implementation in my setup internally with no issues.

st2-dockerfiles # 19 I am blocked here and I would like to hand it over to someone with more experience than me to handle this part. There are lot's of questions but I am sure those are because of my lack of experience in this field and if someone with the right set of skills joins us, we'll have it ready in no time. never too late for Docker 101. Currently builds but not ready since I need to remove the .env file in light of #120

Is anyone interested? you are more than welcome to join. If you have questions about the current stage, I'd be happy to answer any questions.

[arm4b]

@mosn Based on latest commits, that's a good progress towards official st2chatops implementation in K8s/Helm.

We can return anytime to it as you wish and there is no rush. Someone else could fork it and work further or just provide comments/ideas. Additionally, with time we, per our plans can finally hit this task and commit directly to your branch/fork. The bootstrap you did in that PR is already very helpful.

Overall I think it's all good work and valuable, thank you for doing this 👍
We can collectively move that into a good state earlier or later.

[ghost]

The biggest unknown for me how to implement the executable in the dockerfile in such a way that it works. Cannot get /bin/hubot/ to work due to not incorrect or missing PATH. Following is where I stand currently. (Will update repository later today) - installing st2chatops was independent docker container without dependency to st2chatops.env - deploying stackstorm charts with the st2chaops container on top. - this is where chatops pod keeps crashing because of not finding the exec command. I discussed this matter with the community and we concluded staying with the original plan is the right approach. If that’s the case It would be great if we can find support on following 1. In light of the new setup where the st2chatops.env is no longer needed to exists in the docker container, how should the implementation progress? 2. Given that we do not want to have a separate entry point for the st2chatops, what is the best approach towards container initialization? 3. on the helm side of things are their dependencies for st2chatops that we must know of.

…

Sent from my iPhone

On 28. Mar 2019, at 17:42, Eugen C. ***@***.***> wrote: @mosn Based on latest commits, that's good progress towards official st2chatops implementation in K8s/Helm. We can return anytime to it as you wish and there is no rush. Someone else could fork it and work further or just provide comments/ideas. Additionally, with time we, per our plans can finally hit this task and commit directly to your branch/fork. The bootstrap you did in that PR is already very helpful. Overall I think it's all good work and valuable, thank you for doing this 👍 We can collectively move that into a good state earlier or later. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ghost]

@mosn Based on latest commits, that's good progress towards official st2chatops implementation in K8s/Helm.

We can return anytime to it as you wish and there is no rush. Someone else could fork it and work further or just provide comments/ideas. Additionally, with time we, per our plans can finally hit this task and commit directly to your branch/fork. The bootstrap you did in that PR is already very helpful.

Overall I think it's all good work and valuable, thank you for doing this 👍
We can collectively move that into a good state earlier or later.

Absolutely! I'll try to open source as much as I can on our implementation but I am experimenting with Terraform to see if that can make it bit uncluttered for us.
I was working on #6 with auto-DNS and cert-manager but really I still find myself wondering if there's anything to add to help unclutter things. It all boils down on the scale and who StackStrom is good for. a well-automized cluster can really contribute to that image but that's a huge challenge that's not easy to overlook.