Merge 70b9754 into 498080b

open-mmlab · Jan 31, 2023 · aa0355a · aa0355a
2 parents 498080b + 70b9754
commit aa0355a
Show file tree

Hide file tree

Showing 7 changed files with 363 additions and 15 deletions.
diff --git a/docs/en/advanced_tutorials/config.md b/docs/en/advanced_tutorials/config.md
@@ -14,6 +14,8 @@ wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/c
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/my_module.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/optimizer_cfg.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/predefined_var.py
+wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_data_root.py
+wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_num_classes.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/refer_base_var.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_delete_key.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_lr0.01.py
@@ -490,6 +492,98 @@ Config (path: ./example.py): {'model': {'type': 'CustomModel', 'in_channels': [1
 
 :::
 
+### Replace fields with environment variables
+
+When a field is deeply nested, we need to add a long prefix at the command line to locate it. To alleviate this problem, MMEngine allows users to substitute fields in configuration with environment variables.
+
+Before parsing the configuration file, the program will search all `{{$ENV_VAR:DEF_VAL}}` fields and substitute those sections with environment variables. Here, `ENV_VAR` is the name of the environment variable used to replace this section, `DEF_VAL` is the default value if `ENV_VAR` is not set.
+
+When we want to modify the dataset path at the command line, we can take `replace_data_root.py` as an example:
+
+```python
+dataset_type = 'CocoDataset'
+data_root = '{{$DATASET:/data/coco/}}'
+dataset=dict(ann_file= data_root + 'train.json')
+```
+
+If we run `demo_train.py` to parse this configuration file.
+
+```bash
+python demo_train.py replace_data_root.py
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/data/coco/', 'dataset': {'ann_file': '/data/coco/train.json'}}
+```
+
+Here, we don't set the environment variable `DATASET`. Thus, the program directly replaces `{{$DATASET:/data/coco/}}` with the default value `/data/coco/`. If we set `DATASET` at the command line:
+
+```bash
+DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
+```
+
+The value of `data_root` has been substituted with the value of `DATASET` as `/new/dataset/path`.
+
+It is noteworthy that both `--cfg-options` and `{{$ENV_VAR:DEF_VAL}}` allow users to modify fields in command line. But there is a small difference between those two methods. Environment variable substitution occurs before the configuration parsing. If the replaced field is also involved in other fields assignment, the environment variable substitution will also affect the other fields.
+
+We take `demo_train.py` and `replace_data_root.py` for example. If we replace `data_root` by setting `--cfg-options data_root='/new/dataset/path'`:
+
+```bash
+python demo_train.py replace_data_root.py --cfg-options data_root='/new/dataset/path/'
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/data/coco/train.json'}}
+```
+
+As we can see, only `data_root` has been modified. `dataset.ann_file` is still the default value.
+
+In contrast, if we replace `data_root` by setting `DATASET=/new/dataset/path`:
+
+```bash
+DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
+```
+
+Both `data_root` and `dataset.ann_file` have been modified.
+
+Environment variables can also be used to replace other types of fields. We can use `{{'$ENV_VAR:DEF_VAL'}}` or `{{"$ENV_VAR:DEF_VAL"}}` format to ensure the configuration file conforms to python syntax.
+
+We can take `replace_num_classes.py` as an example:
+
+```
+model=dict(
+    bbox_head=dict(
+        num_classes={{'$NUM_CLASSES:80'}}))
+```
+
+If we run `demo_train.py` to parse this configuration file.
+
+```bash
+python demo_train.py replace_num_classes.py
+```
+
+```
+Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 80}}}
+```
+
+Let us set the environment variable `NUM_CLASSES`
+
+```bash
+NUM_CLASSES=20 python demo_train.py replace_num_classes.py
+```
+
+```
+Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 20}}}
+```
+
 ### import the custom module
 
 If we customize a module and register it into the corresponding registry, could we directly build it from the configuration file as the previous [section](#how-to-use-config) does? The answer is "I don't know" since I'm not sure the registration process has been triggered. To solve this "unknown" case, `Config` provides the `custom_imports` function, to make sure your module could be registered as expected.

diff --git a/docs/resources/config/replace_data_root.py b/docs/resources/config/replace_data_root.py
@@ -0,0 +1,3 @@
+dataset_type = 'CocoDataset'
+data_root = '{{$DATASET:/data/coco/}}'
+dataset = dict(ann_file=data_root + 'train.json')
diff --git a/docs/resources/config/replace_num_classes.py b/docs/resources/config/replace_num_classes.py
@@ -0,0 +1 @@
+model = dict(bbox_head=dict(num_classes={{'$NUM_CLASSES:80'}}))
diff --git a/docs/zh_cn/advanced_tutorials/config.md b/docs/zh_cn/advanced_tutorials/config.md
@@ -15,6 +15,8 @@ wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/c
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/optimizer_cfg.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/predefined_var.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/refer_base_var.py
+wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_data_root.py
+wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_num_classes.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_delete_key.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_lr0.01.py
 wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_runtime.py
@@ -490,6 +492,98 @@ Config (path: ./example.py): {'model': {'type': 'CustomModel', 'in_channels': [1
 
 :::
 
+### 使用环境变量替换配置
+
+当要修改的配置嵌套很深时，我们在命令行中需要加上很长的前缀来进行定位。为了更方便地在命令行中修改配置，MMEngine 提供了一套通过环境变量来替换配置的方法。
+
+在解析配置文件之前，MMEngine 会搜索所有的 `{{$ENV_VAR:DEF_VAL}}` 字段，并使用特定的环境变量来替换这一部分。这里 `ENV_VAR` 为替换这一部分所用的环境变量，`DEF_VAL` 为没有设置环境变量时的默认值。
+
+例如，当我们想在命令行中修改数据集路径时，我们可以在配置文件 `replace_data_root.py` 中这样写：
+
+```python
+dataset_type = 'CocoDataset'
+data_root = '{{$DATASET:/data/coco/}}'
+dataset=dict(ann_file= data_root + 'train.json')
+```
+
+当我们运行 `demo_train.py` 来读取这个配置文件时：
+
+```bash
+python demo_train.py replace_data_root.py
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/data/coco/', 'dataset': {'ann_file': '/data/coco/train.json'}}
+```
+
+这里没有设置环境变量 `DATASET`, 程序直接使用默认值 `/data/coco/` 来替换 `{{$DATASET:/data/coco/}}`。如果在命令行前设置设置环境变量则会有如下结果：
+
+```bash
+DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
+```
+
+`data_root` 被替换成了环境变量 `DATASET` 的值 `/new/dataset/path/`。
+
+值得注意的是，`--cfg-options` 与 `{{$ENV_VAR:DEF_VAL}}` 都可以在命令行改变配置文件的值，但他们还有一些区别。环境变量的替换发生在配置文件解析之前。如果该配置还参与到其他配置的定义时，环境变量替换也会影响到其他配置，而 `--cfg-options` 只会改变要修改的配置文件的值。
+
+我们以 `demo_train.py` 与 `replace_data_root.py` 为例。 如果我们通过配置 `--cfg-options data_root='/new/dataset/path'` 来修改 `data_root`：
+
+```bash
+python demo_train.py replace_data_root.py --cfg-options data_root='/new/dataset/path/'
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/data/coco/train.json'}}
+```
+
+从输出结果上看，只有 `data_root` 被修改为新的值。`dataset.ann_file` 依然保持原始值。
+
+作为对比，如果我们通过配置 `DATASET=/new/dataset/path` 来修改 `data_root`:
+
+```bash
+DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
+```
+
+```
+Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
+```
+
+`data_root` 与 `dataset.ann_file` 同时被修改了。
+
+环境变量也可以用来替换字符串以外的配置，这时可以使用 `{{'$ENV_VAR:DEF_VAL'}}` 或者 `{{"$ENV_VAR:DEF_VAL"}}` 格式。`''` 与 `""` 用来保证配置文件合乎 python 语法。
+
+例如，当我们想替换模型预测的类别数时，可以在配置文件 `replace_num_classes.py` 中这样写：
+
+```
+model=dict(
+    bbox_head=dict(
+        num_classes={{'$NUM_CLASSES:80'}}))
+```
+
+当我们运行 `demo_train.py` 来读取这个配置文件时：
+
+```bash
+python demo_train.py replace_num_classes.py
+```
+
+```
+Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 80}}}
+```
+
+当设置 `NUM_CLASSES` 环境变量后：
+
+```bash
+NUM_CLASSES=20 python demo_train.py replace_num_classes.py
+```
+
+```
+Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 20}}}
+```
+
 ### 导入自定义 Python 模块
 
 将配置与注册器结合起来使用时，如果我们往注册器中注册了一些自定义的类，就可能会遇到一些问题。因为读取配置文件的时候，这部分代码可能还没有被执行到，所以并未完成注册过程，从而导致构建自定义类的时候报错。