From f324f8ff0916dd24e82404cb7b73677b5999c854 Mon Sep 17 00:00:00 2001 From: Evandro Myller Date: Wed, 11 Jun 2025 13:10:40 -0300 Subject: [PATCH 1/6] Add make shortcut to run both API and worker --- api/Makefile | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/api/Makefile b/api/Makefile index b177405484bf..ff058ba1b8ff 100644 --- a/api/Makefile +++ b/api/Makefile @@ -97,7 +97,12 @@ django-collect-static: .PHONY: serve serve: poetry run flagsmith start --reload api - + +.PHONY: run-task-processor +serve-with-task-processor: + export TASK_RUN_METHOD="TASK_PROCESSOR" + poetry run python manage.py start --reload --bind 0.0.0.0:8001 task-processor & + poetry run python manage.py start --reload --bind 0.0.0.0:8000 api .PHONY: generate-ld-client-types generate-ld-client-types: From b785426ce5fa8519947f89001f2a9fe3da5ee0f9 Mon Sep 17 00:00:00 2001 From: Evandro Myller Date: Wed, 11 Jun 2025 13:20:48 -0300 Subject: [PATCH 2/6] Update docs --- docs/docs/deployment/hosting/locally-api.md | 39 ++++++++++----------- 1 file changed, 18 insertions(+), 21 deletions(-) diff --git a/docs/docs/deployment/hosting/locally-api.md b/docs/docs/deployment/hosting/locally-api.md index 2cfbd21d5fe7..321a69f63893 100644 --- a/docs/docs/deployment/hosting/locally-api.md +++ b/docs/docs/deployment/hosting/locally-api.md @@ -13,7 +13,7 @@ found in the following section entitled 'Databases'. cd api make install make django-migrate -make serve +make serve-with-task-processor # Or `make serve` for API + synchronous tasks (limited functionality) ``` You can now visit `http:///api/v1/users/config/init/` to create an initial Superuser and @@ -228,7 +228,8 @@ the below variables will be ignored. - `USE_CACHE_FOR_USAGE_DATA`: If enabled, this will use in-process caching to track usage data. Defaults to true. - `API_USAGE_CACHE_SECONDS`: Controls how frequently the usage cache is flushed. Defaults to 60 seconds - `PROMETHEUS_ENABLED`: Enables the Prometheus `/metrics` endpoint. Default is False. -- `PROMETHEUS_HISTOGRAM_BUCKETS`: Allows to specify your bucket sizes for Prometheus histograms, e.g., `"0.5,0.6,1.0,Inf"`. Defaults to Python Prometheus client default histogram sizes. +- `PROMETHEUS_HISTOGRAM_BUCKETS`: Allows to specify your bucket sizes for Prometheus histograms, e.g., + `"0.5,0.6,1.0,Inf"`. Defaults to Python Prometheus client default histogram sizes. #### Security Environment Variables @@ -389,7 +390,7 @@ The application makes use of caching in a couple of locations: and GET /identities endpoints. The application exposes the configuration to allow the caching to be handled in a manner chosen by the developer. The configuration options are explained in more detail below. 5. Environment document - when making heavy use of the environment document, it is often wise to utilise caching to - reduce the load on the database. Details are provided below. + reduce the load on the database. Details are provided below. ### Flags & Identities endpoint caching @@ -426,45 +427,41 @@ cache will be cleared automatically by certain actions in the platform when the | `ENVIRONMENT_CACHE_BACKEND` | Python path to the django cache backend chosen. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `django.core.cache.backends.memcached.PyMemcacheCache` | `django.core.cache.backends.dummy.DummyCache` | | `ENVIRONMENT_CACHE_LOCATION` | The location for the cache. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `127.0.0.1:11211` | `environment-objects` | - ### Environment document caching -When configuring the environment document caching, there are 2 options: persistent or expiring. The expiring cache -will expire after a configurable period. The benefit of this option is that it can be used with all cache -backends, including those without a centralised storage mechanism. Bear in mind that this will mean, however, that -changes will take up to the configured timeout to be reflected in your Flagsmith clients. The persistent cache instead -makes use of an automated process to rebuild the cache whenever a change is made. +When configuring the environment document caching, there are 2 options: persistent or expiring. The expiring cache will +expire after a configurable period. The benefit of this option is that it can be used with all cache backends, including +those without a centralised storage mechanism. Bear in mind that this will mean, however, that changes will take up to +the configured timeout to be reflected in your Flagsmith clients. The persistent cache instead makes use of an automated +process to rebuild the cache whenever a change is made. :::warning -Persistent cache should only be used with cache backends that offer a centralised cache. It should not be used with -e.g. LocMemCache. +Persistent cache should only be used with cache backends that offer a centralised cache. It should not be used with e.g. +LocMemCache. ::: :::info -When using a persistent cache, a change can take a few seconds to update the cache. This can also be optimised by -increasing the performance of your task processor. +When using a persistent cache, a change can take a few seconds to update the cache. This can also be optimised by +increasing the performance of your task processor. ::: - | Environment Variable | Description | Example value | Default | -|---------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------| +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | --------------------------------------------- | | `CACHE_ENVIRONMENT_DOCUMENT_MODE` | The caching mode. One of `PERSISTENT` or `EXPIRING`. Note that although the default is `EXPIRING` there is no caching by default due to the default value of `CACHE_ENVIRONMENT_DOCUMENT_SECONDS` | `PERSISTENT` | `EXPIRING` | | `CACHE_ENVIRONMENT_DOCUMENT_SECONDS` | Number of seconds to cache the environment for (only relevant when `CACHE_ENVIRONMENT_DOCUMENT_MODE=EXPIRING`) | `60` | `0` ( = don't cache) | | `CACHE_ENVIRONMENT_DOCUMENT_BACKEND` | Python path to the django cache backend chosen. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `django.core.cache.backends.memcached.PyMemcacheCache` | `django.core.cache.backends.db.DatabaseCache` | | `CACHE_ENVIRONMENT_DOCUMENT_LOCATION` | The location for the cache. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `127.0.0.1:11211` | `environment-documents` | | `CACHE_ENVIRONMENT_DOCUMENT_OPTIONS` | JSON object representing any additional options required by the specific cache backend. See [here](https://docs.djangoproject.com/en/4.2/topics/cache/#cache-arguments) for further information. | `{"PASSWORD": "securepassword"}` | {} | - - #### Example 1. Expiring local memory cache with 60 second timeout -The following environment variables provide an example for specifying a cache local to each API instance that -expires after 60 seconds. This can be useful in deployments with just a few environments, where there is flexibility -on how long a change to the flags should take to propagate to the clients. +The following environment variables provide an example for specifying a cache local to each API instance that expires +after 60 seconds. This can be useful in deployments with just a few environments, where there is flexibility on how long +a change to the flags should take to propagate to the clients. ``` CACHE_ENVIRONMENT_DOCUMENT_SECONDS: "60" @@ -474,7 +471,7 @@ CACHE_ENVIRONMENT_DOCUMENT_BACKEND: "django.core.cache.backends.locmem.LocMemCa #### Example 2. Persistent redis cache The following environment variables provide an example for specifying a cache, stored in redis, that is automatically -updated whenever a flag is changed. +updated whenever a flag is changed. ``` CACHE_ENVIRONMENT_DOCUMENT_MODE: "PERSISTENT" From 86a969b00ed4a39d17ef32992704d59ef70f49ba Mon Sep 17 00:00:00 2001 From: Evandro Myller <22429+emyller@users.noreply.github.com> Date: Wed, 11 Jun 2025 14:46:30 -0300 Subject: [PATCH 3/6] Break down and compose Co-authored-by: Kim Gustyr --- api/Makefile | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/api/Makefile b/api/Makefile index ff058ba1b8ff..6e7e6d1b6466 100644 --- a/api/Makefile +++ b/api/Makefile @@ -99,10 +99,12 @@ serve: poetry run flagsmith start --reload api .PHONY: run-task-processor -serve-with-task-processor: - export TASK_RUN_METHOD="TASK_PROCESSOR" - poetry run python manage.py start --reload --bind 0.0.0.0:8001 task-processor & - poetry run python manage.py start --reload --bind 0.0.0.0:8000 api +run-task-processor: + poetry run flagsmith start --reload --bind 0.0.0.0:8001 task-processor + +.PHONY: serve-with-task-processor +serve-with-task-processor: TASK_RUN_METHOD=TASK_PROCESSOR +serve-with-task-processor: serve run-task-processor .PHONY: generate-ld-client-types generate-ld-client-types: From 96231023e93639e2ffe75a552a41b83544df1caa Mon Sep 17 00:00:00 2001 From: Evandro Myller Date: Wed, 11 Jun 2025 14:51:15 -0300 Subject: [PATCH 4/6] Revert doc linting --- docs/docs/deployment/hosting/locally-api.md | 37 +++++++++++---------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/docs/docs/deployment/hosting/locally-api.md b/docs/docs/deployment/hosting/locally-api.md index 321a69f63893..73e8629c76e9 100644 --- a/docs/docs/deployment/hosting/locally-api.md +++ b/docs/docs/deployment/hosting/locally-api.md @@ -228,8 +228,7 @@ the below variables will be ignored. - `USE_CACHE_FOR_USAGE_DATA`: If enabled, this will use in-process caching to track usage data. Defaults to true. - `API_USAGE_CACHE_SECONDS`: Controls how frequently the usage cache is flushed. Defaults to 60 seconds - `PROMETHEUS_ENABLED`: Enables the Prometheus `/metrics` endpoint. Default is False. -- `PROMETHEUS_HISTOGRAM_BUCKETS`: Allows to specify your bucket sizes for Prometheus histograms, e.g., - `"0.5,0.6,1.0,Inf"`. Defaults to Python Prometheus client default histogram sizes. +- `PROMETHEUS_HISTOGRAM_BUCKETS`: Allows to specify your bucket sizes for Prometheus histograms, e.g., `"0.5,0.6,1.0,Inf"`. Defaults to Python Prometheus client default histogram sizes. #### Security Environment Variables @@ -390,7 +389,7 @@ The application makes use of caching in a couple of locations: and GET /identities endpoints. The application exposes the configuration to allow the caching to be handled in a manner chosen by the developer. The configuration options are explained in more detail below. 5. Environment document - when making heavy use of the environment document, it is often wise to utilise caching to - reduce the load on the database. Details are provided below. + reduce the load on the database. Details are provided below. ### Flags & Identities endpoint caching @@ -427,41 +426,45 @@ cache will be cleared automatically by certain actions in the platform when the | `ENVIRONMENT_CACHE_BACKEND` | Python path to the django cache backend chosen. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `django.core.cache.backends.memcached.PyMemcacheCache` | `django.core.cache.backends.dummy.DummyCache` | | `ENVIRONMENT_CACHE_LOCATION` | The location for the cache. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `127.0.0.1:11211` | `environment-objects` | + ### Environment document caching -When configuring the environment document caching, there are 2 options: persistent or expiring. The expiring cache will -expire after a configurable period. The benefit of this option is that it can be used with all cache backends, including -those without a centralised storage mechanism. Bear in mind that this will mean, however, that changes will take up to -the configured timeout to be reflected in your Flagsmith clients. The persistent cache instead makes use of an automated -process to rebuild the cache whenever a change is made. +When configuring the environment document caching, there are 2 options: persistent or expiring. The expiring cache +will expire after a configurable period. The benefit of this option is that it can be used with all cache +backends, including those without a centralised storage mechanism. Bear in mind that this will mean, however, that +changes will take up to the configured timeout to be reflected in your Flagsmith clients. The persistent cache instead +makes use of an automated process to rebuild the cache whenever a change is made. :::warning -Persistent cache should only be used with cache backends that offer a centralised cache. It should not be used with e.g. -LocMemCache. +Persistent cache should only be used with cache backends that offer a centralised cache. It should not be used with +e.g. LocMemCache. ::: :::info -When using a persistent cache, a change can take a few seconds to update the cache. This can also be optimised by -increasing the performance of your task processor. +When using a persistent cache, a change can take a few seconds to update the cache. This can also be optimised by +increasing the performance of your task processor. ::: + | Environment Variable | Description | Example value | Default | -| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | --------------------------------------------- | +|---------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|-----------------------------------------------| | `CACHE_ENVIRONMENT_DOCUMENT_MODE` | The caching mode. One of `PERSISTENT` or `EXPIRING`. Note that although the default is `EXPIRING` there is no caching by default due to the default value of `CACHE_ENVIRONMENT_DOCUMENT_SECONDS` | `PERSISTENT` | `EXPIRING` | | `CACHE_ENVIRONMENT_DOCUMENT_SECONDS` | Number of seconds to cache the environment for (only relevant when `CACHE_ENVIRONMENT_DOCUMENT_MODE=EXPIRING`) | `60` | `0` ( = don't cache) | | `CACHE_ENVIRONMENT_DOCUMENT_BACKEND` | Python path to the django cache backend chosen. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `django.core.cache.backends.memcached.PyMemcacheCache` | `django.core.cache.backends.db.DatabaseCache` | | `CACHE_ENVIRONMENT_DOCUMENT_LOCATION` | The location for the cache. See documentation [here](https://docs.djangoproject.com/en/4.2/topics/cache/). | `127.0.0.1:11211` | `environment-documents` | | `CACHE_ENVIRONMENT_DOCUMENT_OPTIONS` | JSON object representing any additional options required by the specific cache backend. See [here](https://docs.djangoproject.com/en/4.2/topics/cache/#cache-arguments) for further information. | `{"PASSWORD": "securepassword"}` | {} | + + #### Example 1. Expiring local memory cache with 60 second timeout -The following environment variables provide an example for specifying a cache local to each API instance that expires -after 60 seconds. This can be useful in deployments with just a few environments, where there is flexibility on how long -a change to the flags should take to propagate to the clients. +The following environment variables provide an example for specifying a cache local to each API instance that +expires after 60 seconds. This can be useful in deployments with just a few environments, where there is flexibility +on how long a change to the flags should take to propagate to the clients. ``` CACHE_ENVIRONMENT_DOCUMENT_SECONDS: "60" @@ -471,7 +474,7 @@ CACHE_ENVIRONMENT_DOCUMENT_BACKEND: "django.core.cache.backends.locmem.LocMemCa #### Example 2. Persistent redis cache The following environment variables provide an example for specifying a cache, stored in redis, that is automatically -updated whenever a flag is changed. +updated whenever a flag is changed. ``` CACHE_ENVIRONMENT_DOCUMENT_MODE: "PERSISTENT" From 70b3a27fa81576c1403425dc074cf6ee77501281 Mon Sep 17 00:00:00 2001 From: Evandro Myller Date: Wed, 11 Jun 2025 15:09:40 -0300 Subject: [PATCH 5/6] Allow the two commands to run in parallel --- api/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/api/Makefile b/api/Makefile index 6e7e6d1b6466..c521a4b28331 100644 --- a/api/Makefile +++ b/api/Makefile @@ -104,7 +104,8 @@ run-task-processor: .PHONY: serve-with-task-processor serve-with-task-processor: TASK_RUN_METHOD=TASK_PROCESSOR -serve-with-task-processor: serve run-task-processor +serve-with-task-processor: + make -j2 serve run-task-processor .PHONY: generate-ld-client-types generate-ld-client-types: From 043a1012d85f1b7ef69cac4e9a790edfedf0ad66 Mon Sep 17 00:00:00 2001 From: Evandro Myller Date: Wed, 11 Jun 2025 15:09:48 -0300 Subject: [PATCH 6/6] Ensure the database is running --- api/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/Makefile b/api/Makefile index c521a4b28331..fb9accc4d810 100644 --- a/api/Makefile +++ b/api/Makefile @@ -95,11 +95,11 @@ django-collect-static: poetry run python manage.py collectstatic --noinput .PHONY: serve -serve: +serve: docker-up poetry run flagsmith start --reload api .PHONY: run-task-processor -run-task-processor: +run-task-processor: docker-up poetry run flagsmith start --reload --bind 0.0.0.0:8001 task-processor .PHONY: serve-with-task-processor