Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

remove redis from code base #510

Merged
merged 2 commits into from Nov 19, 2020
Merged

remove redis from code base #510

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8f3e0d3
remove redis from code base
lexoyo Nov 5, 2020
f8d85f5
fix ut
lexoyo Nov 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
41 changes: 0 additions & 41 deletions Makefile
View file
Open in desktop
Expand Up @@ -61,11 +61,6 @@ rebuild-data: rebuild-data-test rebuild-data-dev
stop-services:
cd docker/ && docker-compose stop

consume-tasks:
huey_consumer --workers=32 --worker-type=thread labonneboite.common.maps.precompute.huey
consume-tasks-dev:
LBB_ENV=development $(MAKE) consume-tasks

# Cleanup
# -------

Expand Down Expand Up @@ -349,42 +344,6 @@ run-importer-job-08-populate-flags:
export LBB_ENV=development && cd labonneboite/importer && python jobs/populate_flags.py


# Redis useful commands
# -----

REDIS_DOCKER_COMPOSE = docker-compose -f docker/docker-compose.yml run redis bash -c
REDIS_CONNECT = redis-cli -h redis -p 6389

redis-get-key:
# Eg: $ redis-get-key KEY="huey.results.huey"
$(REDIS_DOCKER_COMPOSE) "$(REDIS_CONNECT) get '$(KEY)'"

redis-count-keys:
$(REDIS_DOCKER_COMPOSE) "$(REDIS_CONNECT) --scan --pattern '$(PATTERN)' | wc -l"

redis-show-keys:
$(REDIS_DOCKER_COMPOSE) "$(REDIS_CONNECT) --scan --pattern '$(PATTERN)' | head -10"

redis-delete-keys:
# Eg: $ redis-delete-keys PATTERN='"mypattern"'
$(REDIS_DOCKER_COMPOSE) "$(REDIS_CONNECT) --scan --pattern '$(PATTERN)' | tr '\n' '\0' | xargs -L1 -0 $(REDIS_CONNECT) del"

clean-car-isochrone-cache:
@echo '###### Total keys to be deleted \#######'
PATTERN='*isochrone**car*' $(MAKE) redis-count-keys
@echo '###### EXTERMINATE ISOCHRONE CACHE! \######'
$(REDIS_DOCKER_COMPOSE) "$(REDIS_CONNECT) --scan --pattern '*isochrone**car*' | tr '\n' '\0' | xargs -L1 -0 $(REDIS_CONNECT) del"

clean-car-isochrone-and-durations-cache: clean-car-isochrone-cache
@echo '###### Total keys to be deleted \#######'
PATTERN='*durations**car*' $(MAKE) redis-count-keys
@echo '###### EXTERMINATE DURATIONS! \######'
$(REDIS_DOCKER_COMPOSE) "$(REDIS_CONNECT) --scan --pattern '*durations**car*' | tr '\n' '\0' | xargs -L1 -0 $(REDIS_CONNECT) del"

delete-unused-redis-containers:
docker ps -f status=restarting -f name=redis --format "{{.ID}}" \
| xargs docker rm -f

# Test API with key
# -----
# Use:
Expand Down
12 changes: 0 additions & 12 deletions README.md
View file
Open in desktop
Expand Up @@ -167,18 +167,6 @@ MacOS users, if you get a `ld: library not found for -lintl` error when running
The app is available on port `5000` on host machine. Open a web browser, load
http://localhost:5000 and start browsing.

## Run asynchronous tasks

Some parts of the code are run in a separate task queue which can be launched with:

make consume-tasks

Or in development:

make consume-tasks-dev

Asynchronous tasks are backed by Redis and [Huey](https://huey.readthedocs.io/en/latest/).

## Run tests

We are using [Nose](https://nose.readthedocs.io/):
Expand Down
2 changes: 0 additions & 2 deletions labonneboite/common/geocoding/datagouv.py
View file
Open in desktop
Expand Up @@ -157,15 +157,13 @@ def fetch_json(url, name, is_array = False, **params):
response.status_code, response.content
)
# We log an error only if we made an incorrect request
# FIXME Where does this log go? Not found in uwsgi log nor sentry.
log_level = logging.WARNING if response.status_code >= 500 else logging.ERROR
logger.log(log_level, error)
return [] if is_array else {}
try:
result = response.json()
except Exception as e:
error = name + ' responded with an invalid JSON. Error: {}'.format(e)
# FIXME Where does this log go? Not found in uwsgi log nor sentry.
log_level = logging.WARNING if response.status_code >= 500 else logging.ERROR
logger.log(log_level, error)
return [] if is_array else {}
Expand Down
158 changes: 5 additions & 153 deletions labonneboite/common/maps/README.md
View file
Open in desktop
Expand Up @@ -4,7 +4,6 @@ Content:
- [Introduction](#introduction)
- [Settings](#settings)
- [High Level Flow](#high-level-flow)
- [Cache](#cache)
- [Available backends to interact with APIs](#available-backends-to-interact-with-apis)
- [Asynchronous tasks](#asynchronous-tasks)
- [Endpoints](#endpoints)
Expand Down Expand Up @@ -40,8 +39,6 @@ Our project allows searching by two transport modes: car or public transports. T

Requirements:

- **[Huey](https://huey.readthedocs.io/)**: asynchronous task queue used to cache isochrones and durations.
- **A Redis server** (_optional_): used as a backend for Huey. If enabled, the Redis server is also used to cache isochrone results. This is important in production, as it takes a long time to compute a single isochrone. **Default to local cache**.
- **Elastic Search** search by polygon feature is used to find offices.


Expand Down Expand Up @@ -81,22 +78,6 @@ TRAVEL_VENDOR_BACKENDS = {
},
}

# Redis cache (unnecessary if we use local travel cache)
# Useful in production but you need to setup a Redis server!
REDIS_SENTINELS = [] # e.g: [('localhost', 26379)]
REDIS_SERVICE_NAME = 'redis-lbb' # same as declared by sentinel config file
# The following are used only if REDIS_SENTINELS is empty. (useful in
# development where there is no sentinel)
REDIS_HOST = 'localhost'
REDIS_PORT = 6389

# Set this to False to simply trash async tasks (useful in tests)
PROCESS_ASYNC_TASKS = True

# Cache backend
# 'dummy, 'local' or 'redis'
TRAVEL_CACHE = 'local'

# Credentials for fetching travel durations and isochrones
IGN_CREDENTIALS = {
'key': '',
Expand Down Expand Up @@ -140,60 +121,12 @@ Each time he selects an option (transport mode or duration), a new search is mad
### Under the hood

Global view:
1. Precompute isochrone data in a cache.
1. Use this data with Elastic Search to filter offices located inside polygons.

1. Use Elastic Search to filter offices located inside polygons.
1. Display filtered results to the user.
1. Show commuting time for each office in JS.

#### 1/ Precompute isochrone data in a cache.

When loading the view `search.entreprises`, isochrone results are precomputed:


`labonneboite/web/search/views.py`

```
@searchBlueprint.route('/entreprises')
def entreprises():
# ...
# Anticipate future calls by pre-computing duration-related searches
if location:
precompute.isochrones((location.latitude, location.longitude))
```

It means that our app retrieves isochrone data for **each two transport modes** and for **each four durations** specified in `labonneboite/common/maps/constants`.

`labonneboite/common/maps/precompute.py`

```
def isochrones(location):
"""
Compute isochrones asynchronously for all durations and modes. Each isochrone
is computed in a separate task.
"""
for mode in travel.TRAVEL_MODES:
for duration in constants.ISOCHRONE_DURATIONS_MINUTES:
isochrone(location, duration, mode=mode)
```

:warning: Isochrone data is a list of polygons (a list of lists of coordinates), not Elastic Search results! More precisely, its form is:

```
[
[
({latitude}, {longitude}),
({latitude}, {longitude})
],
[...]
]

# Example: `[[(3.504869, 45.910195), (3.504860, 45.910194)], [...]]`
```

:information_source: Data is stored in a cache (see [section Cache](#cache) for further information). The `isochrone` task checks if there is already data for this location in the cache. If not, it stores it.


#### 2/ Use this data with Elastic Search
#### Use Elastic Search

Elastic Search understands pretty well geometric data, including polygons. We just have to pass it to the Elastic Search query. This is made in two steps:

Expand Down Expand Up @@ -223,7 +156,6 @@ def build_json_body_elastic_search(...):
# ...
# Add a filter in Elastic Search
if duration is not None:
# Retrieve cached data
isochrone = travel.isochrone((latitude, longitude), duration, mode=travel_mode)
if isochrone:
for polygon in isochrone:
Expand All @@ -236,8 +168,7 @@ def build_json_body_elastic_search(...):
})
```


#### 3/ Display filtered results to the user
#### Display filtered results to the user

Once results are ready in the `search.entreprises` view, two templates can be shown:
- if request is an AJAX call: `search/results_content.html`
Expand All @@ -258,7 +189,7 @@ Both are located here: `labonneboite/templates/`
:information_source: This is not specific to isochrone requests. It may be useful to refresh results using Ajax calls, for example when a user moves the map with the "refresh results when I move the map" check box enabled.


#### 4/ Show commute time for each office
#### Show commute time for each office

![](readme_images/commuting_time.png)

Expand All @@ -276,64 +207,6 @@ Commute time is not part of former templates, instead it's generated by this scr

:information_source: Durations are retrieved 5 by 5 to avoid timeouts from upstream servers. In theory, the tinier a batch is, the less timeouts we should receive. But tests in production proved that it's not really the case in our hard reality.


## Cache

Retrieving results from third-party APIS is expensive and increases the risk of errors. That's why we use a cache system to store data for a limited time.


### Available caches

Several caches are available, depending on your needs and on your configuration:
- `LocalCache` (default): stores data in memory as a dictionary. :warning: This is highly inefficient and can lead to data loss, so it should not be used in production.
- `DummyCache`: returns None, same as saying "I don't have this data in my cache". Not used for the moment.
- `RedisCache`: stores data in a Redis database. Best option in production but you should have a Redis service available. Note that you can use Redis Sentinel too. Cache auto-expires **30 days** after the last access, as defined in `labonneboite/common/maps/cache.py`.

To switch caches, update the settings:

```
# 'dummy, 'local' or 'redis'
# default to 'local' in development.
TRAVEL_CACHE = 'local'
```

### Cached data

Two kind of data are stored in cache:
- **isochrone**: a list of polygons used to filter offices with Elastic Search.
- **duration**: a list of durations from an origin to multiple destinations. Used to display commute time next to offices details in the search page. Precisely, a list of float values of the same length as `destinations`.

:key: Cached data key follows this format: `[backend_name, func_name, mode] + list(args)`. Example: `["ign", "isochrone", "car", [49.119146, 6.176026], 30]`.

More information here: `labonneboite/common/maps/travel.py`.


### Cleaning cached data

You may want to clear the Redis cache in production or locally to force an update.

#### Locally

Start services to make sure Redis is available.

Then use one of the commands available in the general Makefile depending on your needs (see _Redis useful commands_ section):

```
# ...
# Redis useful commands
# -----

clean-car-isochrone-cache:
# ...
```

#### In production

:information_source: This is only for authorized La Bonne Boite developers as they need access to private files.

Go to our private repository, open the Makefile and look for the 'Redis useful commands' section. Then run the command matching your criteria.


## Available backends to interact with APIs

We use APIs to compute commute duration and to get isochrones. They are located in this folder: `labonneboite/common/maps/vendors`.
Expand Down Expand Up @@ -364,27 +237,6 @@ TRAVEL_VENDOR_BACKENDS = {
}
```


## Asynchronous tasks

Caching isochronous data works hand in hand with asynchronous tasks. We use [Huey](https://huey.readthedocs.io/en/latest/) as a tasks manager and [Redis](https://redis.io) as a backend.

Here is how isochrone precomputing (see [step 1 above](#1-precompute-isochrone-data-in-a-cache)) works:
- Connect to Huey and Backend Cache (Local, Dummy or Redis).
- Define one tasks per travel mode and per duration. This is sent to Huey which runs them in parallel.
- Each task checks whether it's needed to call the API and store new data or not.

Retrieving durations is not done using asynchronous tasks.


:information_source: By default, asynchronous tasks are enabled with a Redis backend. To turn this off and trash tasks, change this in settings:

```
# If false, use DummyHuey as defined here: labonneboite/common/maps/precompute.py#L20
PROCESS_ASYNC_TASKS = False # default in test environment
```


## Endpoints

Two routes are available. They are defined here: `labonneboite/web/maps/views.py`.
Expand Down