Add redis as an (optional) session store backend

[djmaze]

As discussed on the community forum, it is currently a bit of a struggle to make a Zammad work on a multi-machine cluster. The main problem is the use of a shared directory as a kind of message queue for the sessions. Sharing that directory via a distributed filesystem turns out to be really problematic.

A much more suitable tool for this kind of work is Redis. I thus refactored the sessions module to allow for different store backends and implemented a Redis backend in addition to the file backend.

In my local tests, this seems to work well. I still have to try this out in production though.

I am a bit unsure with the naming (is store the right term to use for the backends?). Also, I did not yet have a look at the tests in order to see if or how the new backend can be tested.

In order to use this, you need to set the env variable REDIS_URL to a running Redis server for the Zammad services.

Let me know what you think of this.

[djmaze]

As Github does not want to finish running the tests, I did it all locally. Both with and without a REDIS_URL. In both cases, the following tests failed on my machine:

rspec ./spec/requests/integration/github_spec.rb:91 # GitHub request handling does save ticket issues
rspec ./spec/requests/integration/gitlab_spec.rb:96 # GitLab request handling does save ticket issues
rspec ./spec/requests/proxy_spec.rb:41 # Manage > Settings > System > Network request handling when proxy settings uses invalid config with unknown proxy
rspec ./spec/requests/proxy_spec.rb:51 # Manage > Settings > System > Network request handling when proxy settings uses invalid config with invalid proxy username
rspec ./spec/requests/proxy_spec.rb:61 # Manage > Settings > System > Network request handling when proxy settings uses invalid config with invalid proxy password

The detailed error messages look like they are not related to my changes but maybe to my development setup. (As I was not able to find any documentation on how to setup a development environment, there might be something missing.)

[MrGeneration]

Hey,

maybe this can help you a bit further:
https://docs.zammad.org/en/latest/contributing/ci.html#local

Documentation on this regard currently lacks a bit. Sorry! :-(

[djmaze]

Well, that seems to be outdated. I took the commands out of .github/ci.sh since that is what the CI is currently doing.

[MrGeneration]

Well, that seems to be outdated. I took the commands out of .github/ci.sh since that is what the CI is currently doing.

Not exactly, it's quite fresh actually. ;-x
zammad/zammad-documentation#163

[thorsteneckel]

Hi @djmaze - I'm stunned! Fantastic to receive a PR for this. We're currently right before the 4.0 release so I don't find much time for other things. I just wanted to let you know that these were all tests we added / changed over the last week(s). We haven't had the chance to see the impact on PRs. Looks like there's something broken on our side. Please ignore them for now.
I'll get back to you with more detailed feedback, solutions and a review as fast as I can. This might take 1 or 2 weeks though.

[monotek]

Linking for reference: #1601

[monotek]

In container context i've used memcached for cache_store.
See: https://github.com/zammad/zammad-docker-compose/blob/master/containers/zammad/docker-entrypoint.sh#L58

Could we use redis here too to have a single solution for all file based caches and sessions stores?

[thorsteneckel]

Redis is actually our preferred long term solution 👍

[djmaze]

From my point of view, yes, that would make sense.

[monotek]

Would you mind to extend the pr for that?

[djmaze]

I am not sure that belongs here because this is not about the cache store.

Also, you could probably just replace the :file_store with :redis_store in your docker image similar to how it is currently done with memcached. (Which is of course ugly and should be handled differently.)

Also, switching both cache and session store to Redis might bring a new class of problems, so we should probably also think about how to prevent those.

[thorsteneckel]

I agree. We should limit the scope of this change to a minimum and instead iterate faster with smaller changes.

[djmaze]

If we wanted to take this all the way to Redis, we could probably also improve responsiveness by using redis' blocking list commands instead of polling for updates. But that would be a larger refactoring I'd leave for later.

[mgruner]

Hi @djmaze. The failing tests were caused by missing credentials, that should be fixed now in develop.
Would you be so kind to update your branch to the current state?

[mgruner]

@djmaze FYI, your changes are now in an internal branch and are playing well with the CI. We can take it from here, I believe. More updates soon.

[djmaze]

@mgruner Sorry for not answering quicker. Thanks for taking care! Just tell me if you need help / testing.

[thorsteneckel]

Fantastic work @djmaze - thanks a ton for your contribution and patience 💪 Will be part of Zammad 5

[mgruner]

@djmaze thanks again for this excellent contribution. I merged it more or less unchanged, with some changes to our CI to properly cover this.
This will be a very nice new feature for Zammad 5.0. Thanks again!

[mgruner]

Oh and for the record - I agree that using Memcached for the Rails cache is probably the better tool for the purpose. I will make sure this will be configurable and documented in a nice way for Zammad 5.0 too.