Setting up Caching

[kingo55]

Caravel's own documentation is non-existant but it suggests passing a JSON into the config like so, based on the Flask Cache:

CACHE_CONFIG = {
    'CACHE_TYPE': 'RedisCache',
    'CACHE_REDIS_HOST': '10.0.0.8',
    'CACHE_REDIS_PORT': '6379',
    'CACHE_KEY_PREFIX': '/caravel'
}

Via apache/superset#390

Therefore, I'd assume this should work for the docker:

$ docker run --detach --name caravel -e SECRET_KEY="blah" \
> -e SQLALCHEMY_DATABASE_URI="mysql://blah/caravel_prod" \
> -e CARAVEL_CACHE_CONFIG=\{CACHE_DEFAULT_TIMEOUT:600,CACHE_KEY_PREFIX:'caravel',CACHE_REDIS_HOST:'blah',CACHE_REDIS_PORT:'6379',CACHE_TYPE:'redis'\} \
> -e CARAVEL_MAPBOX_API_KEY=blah \
> --publish 8088:8088 amancevice/caravel

But it throws this when I start the container and check the logs:

$ docker logs caravel
/usr/lib/python3.5/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.script is deprecated, use flask_script instead.
  .format(x=modname), ExtDeprecationWarning
/usr/lib/python3.5/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.sqlalchemy is deprecated, use flask_sqlalchemy instead.
  .format(x=modname), ExtDeprecationWarning
Traceback (most recent call last):
  File "/usr/bin/caravel", line 15, in <module>
    import caravel
  File "/usr/lib/python3.5/site-packages/caravel/__init__.py", line 25, in <module>
    app.config.from_object(CONFIG_MODULE)
  File "/usr/lib/python3.5/site-packages/flask/config.py", line 163, in from_object
    obj = import_string(obj)
  File "/usr/lib/python3.5/site-packages/werkzeug/utils.py", line 418, in import_string
    __import__(import_name)
  File "/usr/lib/python3.5/site-packages/caravel/config.py", line 186, in <module>
    CACHE_DEFAULT_TIMEOUT = CACHE_CONFIG.get('CACHE_DEFAULT_TIMEOUT')
AttributeError: 'str' object has no attribute 'get'

EDIT: I'd be happy to do a pull request with further instructions setting up Redis Caching.

[amancevice]

This looks like a slew of problems that unfolded one after another for me:

CACHE_CONFIG

The CACHE_CONFIG variable is expected to be a dict and environment variables are not capable of representing that. The workaround right now is to maintain and lay down your own caravel_config.py file.

Alternatively, I can add some code to the default caravel_config.py that tries to parse the CACHE_CONFIG environment variable as JSON data. That seems like a reasonable thing for me to do, but:

Redis is not installed

The redis pip is not installed in the amancevice/caravel image by default. I'd need to add this to the image or you'd need to roll your own Dockerfile. Something like

FROM amancevice/caravel:0.10.0
RUN pip3 install redis==2.10.5

Ought to do the trick. I don't think it would be a huge burden to bake this into the standard image but I'd need to think about it. Regardless, we move onto the next issue:

Incorrect Config

The configuration you provided seems to be incorrect. Specifically the 'CACHE_TYPE': 'RedisCache' keypair should be 'CACHE_TYPE': 'redis'. This is according to the ticket you linked near the end of the thread.

After all that, though:

Python3 is not fully supported in 0.10.0

I got it mostly working except that it looks like there is a bug in caravel 0.10.0 that's fixed on the HEAD of master. These lines were added after the release of 0.10.0 and appear to resolve the issue (though I haven't confirmed this). So it looks like we will need to wait for the next version of caravel to come out for this to be supported.

[amancevice]

For posterity here is what I used to test this stuff:

I created a new empty caravel DB

touch caravel.db

I wrote a docker-compose.yml file:

version: '2'
services:
  redis:
    image: redis:alpine
  caravel:
    image: amancevice/caravel
    depends_on:
    - redis
    ports:
    - "8088:8088"
    volumes:
    - "./caravel.db:/home/caravel/db/caravel.db"
    links:
    - redis
    environment:
    - "DEBUG=1"
    - "CACHE_CONFIG={\"CACHE_TYPE\": \"redis\", \"CACHE_DEFAULT_TIMEOUT\": 30, \"CACHE_KEY_PREFIX\": \"caravel_\", \"CACHE_REDIS_HOST\": \"redis\", \"CACHE_REDIS_PORT\": 6379, \"CACHE_REDIS_DB\": 1, \"CACHE_REDIS_URL\": \"redis://redis:6379/1\"}"

I brought up the composition with docker-compose up -d and then created my demo db with:

docker-compose exec caravel demo

I then had a DB to work with as I experimented with the setup.

[amancevice]

I have added a amancevice/caravel:redis image that is built off of Python2 and has the redis pip installed. Let me know if this works for you. Remember to change your CACHE_CONFIG environment variable to use "CACHE_TYPE": "redis"

[kingo55]

Ah yes... not as easy as I first imagined. Not a huge deal - may just have to use this image as a base as suggested. Good excuse to learn Docker!

Redis and Cache config sound like the bigger problems - at least Py 3 support is becoming standard pretty quickly.

I'll let you know how amancevice/caravel:redis goes.

[PanJ]

I tested amancevice/caravel:redis and it works great! I think you should merge it to latest branch. Separate caching environment (not in dictionary) would be great.

[amancevice]

@PanJ I'm hoping AirBnB (paging @mistercrunch :) releases the next version of caravel soon since it contains fixes for Python3. I'm not crazy about reverting the project back to Python2 but I would be okay with adding the redis pip to the image.

[amancevice]

I updated the to 0.11.0 and added the redis pip to the amancevice/caravel:0.11.0 tag. I confirmed I got redis caching to work so I am going to close this issue but feel free to re-open if you have problems.