Added kubernetes yaml files #58

kersten · 2017-10-24T13:59:25Z

A first set of yaml files for kubernetes.

monotek · 2017-10-24T21:42:40Z

Question: why do you use Zammad nginx container?

Imho the kubernetes ingress controller could do the proxy work too.

kersten · 2017-10-24T21:48:11Z

I just copied the docker setup over to kubernetes.

Is the kubernetes Ingress able to mount the pvc and use a custom config with wss?

Another question would be, eg: Is the tectonic ingress or some other ingresses able to do that too?

kersten · 2017-10-24T21:54:43Z

Another thing: I don't know the zammad core, would it be possible to use nginx websocket upstream functionality instead of the custom wss port?

monotek · 2017-10-24T21:57:27Z

I'm not sure that all ingress controllers support it but at least nginx & traefik ingress controlles can do it:

Nginx example: https://github.com/nginxinc/kubernetes-ingress/blob/master/examples/websocket/README.md

kersten · 2017-10-24T21:59:57Z

Ok, I will have a look at it, but that's stuff for tomorrow or so. I have to see.

monotek · 2017-10-24T22:05:53Z

No hurry!
Thanks for your work :-)

kersten · 2017-10-25T10:06:15Z

This one is now working as expected. Let me know if you have another wish :)

kersten · 2017-10-25T13:00:31Z

Maybe I could add a couple more Readme.md stuff.

At least tell people that the need ReadWriteMany persistent volumes. This can be delivered from glusterfs or storageos.

monotek · 2017-10-28T13:10:37Z

Google cloud test failed :-(

Had to remove the s3 & cronjob yml files to get through "kubectl apply -f ." but containers wont start because of errors like:

warning PersistentVolumeClaim is not bound: "zammad-home" (repeated 3 times) default-scheduler
warning Failed to provision volume with StorageClass "standard": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce ReadOnlyMany] are supported

I'll try with minkube later.

monotek · 2017-10-29T12:21:11Z

It also did not work out of the box on minikube.

GCE was not so much of a suprise. Maybe it works by adding some special storage container (nfs, or glusterfs). But i don't have enough time at the moment to try it out.

Did you tried it somewhere else as in your environment? At least on Minikube it should work out of the box, to be able to test it.

kersten · 2017-10-29T13:07:16Z

Hi,

this is correct. For Zammad you will need ReadWriteMany disks. This is possible using eg GlusterFs or storageos. But that setup would be overkill to list here and hardly to maintain. As I said, maybe the documentation needs to be a bit more clear.

I never tried minikube, so I don't know if we could get this working out of the box.

monotek · 2017-10-29T13:10:57Z

As minikube is the official kubernetes testing environment we should try to get it work there. This should also be generic enough to use it as a starting point to get it work in other environments.

monotek · 2017-10-31T09:08:00Z

Update....
Got it running on Minikube, by starting with a fresh installation. Seems my Minikube installation was to old or fucked up in some way. I'll push soon with some changes...

monotek · 2017-10-31T10:06:14Z

After a while the websocket and scheduler containers crashed.

PersistentVolumeClaim is not bound: "zammad-home"
Back-off restarting failed container
Error syncing pod

Pod log of scheduler and websocket server look like:

scheduler can access raillsserver now...
Bundler::GemNotFound: Could not find rake-12.0.0 in any of the sources
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/spec_set.rb:87:in `block in materialize'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/spec_set.rb:81:in `map!'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/spec_set.rb:81:in `materialize'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/definition.rb:159:in `specs'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/definition.rb:218:in `specs_for'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/definition.rb:207:in `requested_specs'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/runtime.rb:109:in `block in definition_method'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/runtime.rb:21:in `setup'
  /usr/local/lib/ruby/gems/2.4.0/gems/bunhungen für dieses Kino möglichdler-1.15.4/lib/bundler.rb:101:in `setup'
  /usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/setup.rb:19:in `<top (required)>'
  /usr/local/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  /usr/local/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
bundler: failed to load command: script/scheduler.rb (script/scheduler.rb)

I guess its still a problem that the all containers try to write to zammads tmp dir.

I tend to create kubernetes containers which use nfs or glusterfs to share a network directory. Also trying to figure out if redis, memcached or something similar can be used for tmp dir replacement.

This is my current progress: https://github.com/monotek/zammad-docker-compose/tree/beyond-agentur-master

kersten · 2017-11-01T06:06:30Z

Hi, yes this sounds like a volume problem. I use glusterfs as ReadWriteMany volumes. I had that error too, restarting railsserver helped. But that would not be the solution.

We will get this working soon, I hope :)

kersten · 2017-11-01T06:08:11Z

The removal of S3 is correct I think. It's something like glusterfs that should not be handled here. I think its not working because you didn't pass correct config values.

monotek · 2017-11-01T14:41:34Z

We'll evaluate to use memcached like described in: zammad/zammad#1601

Geo719 · 2017-11-04T10:09:01Z

Hi,
I pulled the docker-compose version of Zammad. Next I tried running kompose convert / kompose up after altering the docker-compose.yml version from 3.3 to 3.

The system starts up but there are problems:

After kompose up:
FATA Error while deploying application: persistentvolumeclaims "data-zammad" already exists
Along with this the zammad-nginx pod gives following errors:

PersistentVolumeClaim is not bound: "data-zammad"
Back-off restarting failed container
Error syncing pod

zammad-nginx exits with error because of:
nginx: [emerg] host not found in upstream "zammad-railsserver:3000" in /etc/nginx/conf.d/zammad.conf:8

any clues?

monotek · 2017-11-04T10:11:54Z

It does not work, without shared storage.
Thats the reason its not merged.

We're currently evaluating to use memcached to fix the problem. See: zammad/zammad#1601
Vote for the issue, if you're interested.

Geo719 · 2017-11-04T10:28:28Z

Yes, I did read - maybe I did not understand.
So minikube does not "support" ReadWriteMany.
Would it be an option to use NFS?
minikube start --feature-gates=DynamicVolumeProvisioning=false
and then hook on a NFS?

Beside, why using docker-compose v3.3? There are no v3.3 specific rules?!

Geo719 · 2017-11-04T10:52:59Z

monotek, I had not seen the issue #1601
Thanks for the hint

PS: are you aware of minikube-nfs?
Wouldn´t that a more "production" like setup instead of using memcached?

monotek · 2017-11-04T11:09:21Z

Yes, NFS, Glusterfs or whatever network filesystem would work. Unfortunately it's kind of overhead to add this to Zammad, as additional containers (storage server) and changes to existing ones (storage clients) would be needed.

That's the reason we thinking about to implement memcached instead, as parts of Zammad (cache) would already support it.

Unfortunately we don't have a schedule for that at the moment. So voting for the linked issue could help speed up things a bit ;-)

Can't remember the reason for compose v3.3. Maybe there is none but it's offtopic anyway.

Minikube NFS is not a solution. It should be possible to run Zammad in any kubernetes cloud. Not just minikube.

Geo719 · 2017-11-04T12:02:52Z

OK, I think I got it and already voted :)

monotek · 2017-11-21T18:22:56Z

I just merged your changes to master.
Many thanks again :-)
Also added a NFS server that serves 1 GB tmpfs dir to fix the volume issues.

Added kubernetes yaml files

3efca6c

kersten mentioned this pull request Oct 24, 2017

Add kubernetes example #3

Closed

kersten added 3 commits October 24, 2017 22:42

Added zammad-backup und s3 sync

5a8fbbd

Added backup once

b6b0b05

removed double code

69a3ed8

kersten added 2 commits October 25, 2017 11:48

added port for wss and 50m upload file size

67af06f

Proxy websocket from kubernetes ingress directly to zammand-websocket

1e2cc84

monotek closed this Nov 21, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added kubernetes yaml files #58

Added kubernetes yaml files #58

kersten commented Oct 24, 2017

monotek commented Oct 24, 2017 •

edited

kersten commented Oct 24, 2017 •

edited

kersten commented Oct 24, 2017

monotek commented Oct 24, 2017

kersten commented Oct 24, 2017

monotek commented Oct 24, 2017

kersten commented Oct 25, 2017

kersten commented Oct 25, 2017

monotek commented Oct 28, 2017

monotek commented Oct 29, 2017

kersten commented Oct 29, 2017

monotek commented Oct 29, 2017

monotek commented Oct 31, 2017

monotek commented Oct 31, 2017 •

edited

kersten commented Nov 1, 2017

kersten commented Nov 1, 2017

monotek commented Nov 1, 2017

Geo719 commented Nov 4, 2017 •

edited

monotek commented Nov 4, 2017 •

edited

Geo719 commented Nov 4, 2017

Geo719 commented Nov 4, 2017 •

edited

monotek commented Nov 4, 2017 •

edited

Geo719 commented Nov 4, 2017

monotek commented Nov 21, 2017 •

edited

Added kubernetes yaml files #58

Added kubernetes yaml files #58

Conversation

kersten commented Oct 24, 2017

monotek commented Oct 24, 2017 • edited

kersten commented Oct 24, 2017 • edited

kersten commented Oct 24, 2017

monotek commented Oct 24, 2017

kersten commented Oct 24, 2017

monotek commented Oct 24, 2017

kersten commented Oct 25, 2017

kersten commented Oct 25, 2017

monotek commented Oct 28, 2017

monotek commented Oct 29, 2017

kersten commented Oct 29, 2017

monotek commented Oct 29, 2017

monotek commented Oct 31, 2017

monotek commented Oct 31, 2017 • edited

kersten commented Nov 1, 2017

kersten commented Nov 1, 2017

monotek commented Nov 1, 2017

Geo719 commented Nov 4, 2017 • edited

monotek commented Nov 4, 2017 • edited

Geo719 commented Nov 4, 2017

Geo719 commented Nov 4, 2017 • edited

monotek commented Nov 4, 2017 • edited

Geo719 commented Nov 4, 2017

monotek commented Nov 21, 2017 • edited

monotek commented Oct 24, 2017 •

edited

kersten commented Oct 24, 2017 •

edited

monotek commented Oct 31, 2017 •

edited

Geo719 commented Nov 4, 2017 •

edited

monotek commented Nov 4, 2017 •

edited

Geo719 commented Nov 4, 2017 •

edited

monotek commented Nov 4, 2017 •

edited

monotek commented Nov 21, 2017 •

edited