-
Notifications
You must be signed in to change notification settings - Fork 171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set up Airflow Variable defaults with descriptions automatically #4202
Comments
Hi @AetherUnbound, this looks like something I would like to take on. Secondly, where exactly would the addition of the description be done, In the in-memory loaded JSON or in the source JSON file? |
Hey @madewithkode! I think that step would necessarily need to be completed by the maintainers. Once we have the Variables documented, we can go in one by one for a one-off update of all the existing Variables to add descriptions. So please do feel free to take on the code work for this PR, and we'll handle the description adding in production once it's ready! |
Noted. Would open a draft PR and let you know once I come up with something. |
Hi @AetherUnbound does the following JSON file snippet mirror the goal of this task?
|
Alright, so I tried to do some digging on what the JSON file actually looked like, and wasn't able to find any in the documentation linked above. Instead, I added a variable locally then ran the following: $ airflow@ef2c68b81811:/opt/airflow$ airflow variables export -
{
"SAMPLE_VARIABLE": "{\"sample_key\": true, \"other_key: 1, \"final_key\": [\"a\"]}"
}
1 Variables successfully exported. So...even though this Variable has a description, there's no way to export/import that description using JSON files and However, it does look like we can use $ airflow@ef2c68b81811:/opt/airflow$ airflow variables set --help
Usage: airflow variables set [-h] [--description DESCRIPTION] [-j] [-v] key VALUE
Set variable
Positional Arguments:
key Variable key
VALUE Variable value
Options:
-h, --help show this help message and exit
--description DESCRIPTION
Variable description, optional when setting a variable
-j, --json Serialize JSON variable
-v, --verbose Make logging output more verbose The only trouble here is that there aren't any options for how to handle existing Variables, which leads me to believe it'll automatically overwrite if this command is run. We can determine that first though with $ airflow@ef2c68b81811:/opt/airflow$ airflow variables list -o plain
key
SAMPLE_VARIABLE
OTHER So here's what I think are the next steps:
This is all admittedly way more complicated than initially expected @madewithkode 😅 And before we go with that approach, I want to make sure it makes sense with @stacimc! |
Wow -- interesting find. Disappointing we can't import them so easily, but the next steps @AetherUnbound listed do make sense to me, and I think this feature is very much worth it! |
Hi @AetherUnbound nice find and recommendations, I think this is still well within my capacity 😅, would sure be an interesting problem to tackle. I'd await your comment on what the default values and description for each of the variables would look like. |
@madewithkode I've created the TSV (it's saved as a CSV here because GitHub doesn't accept TSV uploads directly). It is using a tab for delimiting the columns. Note that the description field has double quotes ( |
Just following up to note that I've added the descriptions to production as well 🚀 |
Description
We have a number of Airflow Variables which act as levers for some of our DAGs, allowing us to tweak settings for certain DAGs without having to restart the stack. At present, the documentation for them is spread out and not easily accessible:
SILENCED_SLACK_NOTIFICATIONS
SKIPPED_INGESTION_ERRORS
CONFIGURATION_OVERRIDES
For this (and any other Variables that it might seem useful for), we should set up a JSON file with a default value and an adequate description for both what the shape of the expected values are and how this affects the related DAGs. This can effectively serve as documentation for the Variables above.
We can then use
airflow variables import
(with the--action-on-existing-key
set toskip
) as part of ourentrypoint.sh
to ensure it runs both locally and in production on every startup. In production, if the keys already exists, we'll just need to add the descriptions to them once since we won't be overwriting the values.The text was updated successfully, but these errors were encountered: