Custom data set training using pre-trained weights

[alrightkami]

Hi there, thank you so much for your hard work for release.
Could you please provide a tutorial for custom data training using pre-trained weights?
And is it possible at the moment to fine-tune DINO with Swin backbones?

[rentainhe]

Hi there, thank you so much for your hard work for release. Could you please provide a tutorial for custom data training using pre-trained weights? And is it possible at the moment to fine-tune DINO with Swin backbones?

Hi! We will release some converted or detrex pretrained DINO-Swin weights recently, and the tutorials for custom datasets will be coming soon. You can also check detectron2 custom dataset tutorials here: documentation

[alrightkami]

@rentainhe thank you for the fast reply! I have a bit of experience with detectron2 already, is the usage of detrex the same? in detectron2 I would do something like this to train:

register_coco_instances()
cfg = get_cfg()
cfg.merge_from_file()  
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url()
# etc.
trainer = DefaultTrainer(cfg)
trainer.train()

or did you mean I only need to register data?

[rentainhe]

@rentainhe thank you for the fast reply! I have a bit of experience with detectron2 already, is the usage of detrex the same? in detectron2 I would do something like this to train:

register_coco_instances()
cfg = get_cfg()
cfg.merge_from_file()  
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url()
# etc.
trainer = DefaultTrainer(cfg)
trainer.train()

or did you mean I only need to register data?

Yes, we use the same training engine as detectron2. But we've adopted the newly designed LazyConfig config system which we believe will work better than the old yacs config system.

After registering your own datasets, maybe you can train DETR on your custom datasets as follows:

1. Modify dataloader configs here: data_detr.py, if your datasets have the same format as COCO, I guess you can directly use the default API provided in detectron2, e.g., get_detection_dataset_dicts, build_detection_test_loader. Or you may need to write your own dataset_mapper follows DetrDatasetMapper.
2. Import your new dataloader config in dab_detr_r50_50ep.py, using DAB-DETR as an example.

This way may take you some time to learn the dataloader API designs in detectron2. However, with the help of LazyConfig, there's another way for you to use your own datasets.

You can hack into train_net.py and implement your own datasets and dataloader there. As you can see, in train_net.py, we directly pass the instantiated dataloader into Trainer. Maybe you can instantiate your own dataloader here

# example

# train_loader = instantiate(cfg.dataloader.train)
train_loader = your_own_train_loader

[rentainhe]

@rentainhe thank you for the fast reply! I have a bit of experience with detectron2 already, is the usage of detrex the same? in detectron2 I would do something like this to train:

register_coco_instances()
cfg = get_cfg()
cfg.merge_from_file()  
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url()
# etc.
trainer = DefaultTrainer(cfg)
trainer.train()

or did you mean I only need to register data?

To put it simply, register datasets -> create your own dataset mapper if needed -> update data config

There are also some discussions on custom datasets in detectron2 here

[rentainhe]

@rentainhe thank you for the fast reply! I have a bit of experience with detectron2 already, is the usage of detrex the same? in detectron2 I would do something like this to train:

register_coco_instances()
cfg = get_cfg()
cfg.merge_from_file()  
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url()
# etc.
trainer = DefaultTrainer(cfg)
trainer.train()

or did you mean I only need to register data?

If you are not in a rush, we will provide tutorials about training on custom datasets as soon as possible.

And if you have any other questions, leave an issue anytime if you need our help.

[ray-lee-94]

@rentainhe
When will you release the fully trained DINO pre-trained weights? I can only find the 12-epoch ones.

[rentainhe]

@rentainhe When will you release the fully trained DINO pre-trained weights? I can only find the 12-epoch ones.

We will release DINO training results in the future version if you're not in a rush. We will also provide the converter for the users to convert the weight trained from the original repo later, maybe you can use the pretrained 24 and 36epochs results by converting them from the official repo. @VCBE123

[alrightkami]

@rentainhe hey there! I managed to start a custom training with the new DINO weights you guys released yesterday; thank you so much for that!
However, I couldn't figure a way to start a hard-negatives training i.e. do NOT filter empty annotations out.

Now while training I'm getting the following message:
[10/19 15:45:10 d2.data.build]: Removed 1194 images with no usable annotations. 821 images left.

I added the next line in the config.py I use for training following the way you would do it for Detectron2:
dataloader.filter_empty_annotations= False
but it did not change anything.

Your help would be much appreciated!

[rentainhe]

@rentainhe hey there! I managed to start a custom training with the new DINO weights you guys released yesterday; thank you so much for that! However, I couldn't figure a way to start a hard-negatives training i.e. do NOT filter empty annotations out.

Now while training I'm getting the following message: [10/19 15:45:10 d2.data.build]: Removed 1194 images with no usable annotations. 821 images left.

I added the next line in the config.py I use for training following the way you would do it for Detectron2: dataloader.filter_empty_annotations= False but it did not change anything.

Your help would be much appreciated!

Hello @alrightkami , you can set the config as follows to not filter empty annotations:

dataloader.train.dataset.filter_empty = False

You can refer to dino_r50_4scale_24ep.py for more details.

[alrightkami]

@rentainhe Thank you so much! It worked. However, I have another issue now with inference. Could you please provide an example of custom image inference for the fine-tuned weights?
I'm trying to follow this issue, but it's not working out well so far. I'm getting an error message while trying to instantiate a LazyConfig as suggested here:

---> [28] cfg = get_config('/home/data/detrex/output/2110_0730/config.yaml')
NameError: name 'get_config' is not defined

Update: things I tried
1) I was trying to find where the get_config() comes from and finally figured it out:
cfg = model_zoo.get_config('/home/data/detrex/output/2110_0730/config.yaml')

However, it's expecting the config file to be from Detectron2's ModelZoo:
RuntimeError: /home/jovyan/data/kamila/detrex/output/graffiti2/2110_0730/config.yaml not available in Model Zoo!

2) I also looked into the model_zoo.py code and tried to import config as follows:

cfg_path = '/home/data/detrex/output/2110_0730/config.yaml'
cfg = get_cfg(cfg_path)

but got:
TypeError: get_cfg() takes 0 positional arguments but 1 was given

3) LazyConfig also doesn't work here for me:

cfg = LazyConfig.load(cfg_path)

I'm getting:

ConstructorError: while constructing a Python object
cannot find module 'None' (No module named 'None')
  in "/home/data/detrex/output/2110_0730/config.yaml", line 103, column 106

4) When I'm trying to use detrex's config:

from detrex.config import get_config
cfg = get_config("common/data/coco_detr_graffiti.py")
cfg.train.init_checkpoint = trained_model 

I'm also getting an error:

----> [9] cfg.train.init_checkpoint = trained_model
ConfigAttributeError: Missing key train
    full_key: train
    object_type=dict

[rentainhe]

hello @alrightkami ! The get_config() functions will automatically get the config from configs/common after building detrex.

the configs in configs/common will be soft linked to detrex/config

you need to add your config here~, then you can load your own config file by using get_config(), however, the better way is to directly add a new config.py in your projects.

I'm reproducing your problems now and try to solve it~

And I think you can open a new issue about this problem which can make more people see this issue~, maybe others can also help you to solve this problem.

[alrightkami]

@rentainhe Thank you for trying to help! I still wasn't able to fix the issue today and run inference.
Could you please provide a code snippet of how to do inference for fine-tuned weights on your own data?
This would be very helpful.

[rentainhe]

@rentainhe Thank you so much! It worked. However, I have another issue now with inference. Could you please provide an example of custom image inference for the fine-tuned weights? I'm trying to follow this issue, but it's not working out well so far. I'm getting an error message while trying to instantiate a LazyConfig as suggested here:

---> [28] cfg = get_config('/home/data/detrex/output/2110_0730/config.yaml')
NameError: name 'get_config' is not defined

Update: things I tried 1) I was trying to find where the get_config() comes from and finally figured it out: cfg = model_zoo.get_config('/home/data/detrex/output/2110_0730/config.yaml')

However, it's expecting the config file to be from Detectron2's ModelZoo: RuntimeError: /home/jovyan/data/kamila/detrex/output/graffiti2/2110_0730/config.yaml not available in Model Zoo!

2) I also looked into the model_zoo.py code and tried to import config as follows:

cfg_path = '/home/data/detrex/output/2110_0730/config.yaml'
cfg = get_cfg(cfg_path)

but got: TypeError: get_cfg() takes 0 positional arguments but 1 was given

3) LazyConfig also doesn't work here for me:

cfg = LazyConfig.load(cfg_path)

I'm getting:

ConstructorError: while constructing a Python object
cannot find module 'None' (No module named 'None')
  in "/home/data/detrex/output/2110_0730/config.yaml", line 103, column 106

4) When I'm trying to use detrex's config:

from detrex.config import get_config
cfg = get_config("common/data/coco_detr_graffiti.py")
cfg.train.init_checkpoint = trained_model 

I'm also getting an error:

----> [9] cfg.train.init_checkpoint = trained_model
ConfigAttributeError: Missing key train
    full_key: train
    object_type=dict

for the 4-th one, you can try to use:

from detrex.config import get_config
cfg = get_config("common/data/coco_detr_graffiti.py").train
cfg.train.init_checkpoint = trained_model 

And I'm testing other case now~

[alrightkami]

@rentainhe I tried it, but it didn't change anything, I'm still getting the same error

[alrightkami]

you need to add your config here~, then you can load your own config file by using get_config(), however, the better way is to directly add a new config.py in your projects.

FYI: I added the config file in the configs/common, but it's not a pretty solution because it causes the problem with imports:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/tmp/ipykernel_58675/3920766154.py in <module>
      1 path_to_imgs = "..."
      2 output_folder = "..."
----> 3 draw_labels(path_to_imgs, output_folder)

/tmp/ipykernel_58675/1565024436.py in draw_labels(path_to_imgs, output_folder, save_originals)
     27     # initialise configuration (has to be the same as in the training)
     28     # cfg = get_config("common/data/coco_detr_graffiti.py").train
---> 29     cfg = get_config("common/data/dino_swin_large_384_4scale_12ep_graffiti.py")
     30     cfg.train.init_checkpoint = trained_model
     31     model = instantiate(cfg.model)

/home/jovyan/data/kamila/detrex/detrex/config/config.py in get_config(config_path)
     50     if not os.path.exists(cfg_file):
     51         raise RuntimeError("{} not available in detrex configs!".format(config_path))
---> 52     cfg = LazyConfig.load(cfg_file)
     53     return cfg

/home/jovyan/data/kamila/detrex/detectron2/detectron2/config/lazy.py in load(filename, keys)
    209                 # 1. make filename appears in stacktrace
    210                 # 2. make load_rel able to find its parent's (possibly remote) location
--> 211                 exec(compile(content, filename, "exec"), module_namespace)
    212 
    213             ret = module_namespace

/home/jovyan/data/kamila/detrex/detrex/config/configs/common/data/dino_swin_large_384_4scale_12ep_graffiti.py in <module>
      1 import datetime
      2 from detrex.config import get_config
----> 3 from .models.dino_swin_large_384 import model
      4 # from detrex.projects.dino.configs.models.dino_swin_large_384 import model
      5 

/home/jovyan/data/kamila/detrex/detectron2/detectron2/config/lazy.py in new_import(name, globals, locals, fromlist, level)
    138             and (globals.get("__package__", "") or "").startswith(_CFG_PACKAGE_NAME)
    139         ):
--> 140             cur_file = find_relative_file(globals["__file__"], name, level)
    141             _validate_py_syntax(cur_file)
    142             spec = importlib.machinery.ModuleSpec(

/home/jovyan/data/kamila/detrex/detectron2/detectron2/config/lazy.py in find_relative_file(original_file, relative_import_path, level)
    125             cur_file += ".py"
    126         if not PathManager.isfile(cur_file):
--> 127             raise ImportError(
    128                 f"Cannot import name {relative_import_path} from "
    129                 f"{original_file}: {cur_file} has to exist."

ImportError: Cannot import name models.dino_swin_large_384 from /home/jovyan/data/kamila/detrex/detrex/config/configs/common/data/dino_swin_large_384_4scale_12ep_graffiti.py: /home/jovyan/data/kamila/detrex/detrex/config/configs/common/data/models/dino_swin_large_384.py has to exist.

I fixed it when I changed from .models.dino_swin_large_384 import model
to from projects.dino.configs.models.dino_swin_large_384 import model

[rentainhe]

you need to add your config here~, then you can load your own config file by using get_config(), however, the better way is to directly add a new config.py in your projects.

FYI: I added the config file in the configs/common, but it's not a pretty solution because it causes the problem with imports:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/tmp/ipykernel_58675/3920766154.py in <module>
      1 path_to_imgs = "..."
      2 output_folder = "..."
----> 3 draw_labels(path_to_imgs, output_folder)

/tmp/ipykernel_58675/1565024436.py in draw_labels(path_to_imgs, output_folder, save_originals)
     27     # initialise configuration (has to be the same as in the training)
     28     # cfg = get_config("common/data/coco_detr_graffiti.py").train
---> 29     cfg = get_config("common/data/dino_swin_large_384_4scale_12ep_graffiti.py")
     30     cfg.train.init_checkpoint = trained_model
     31     model = instantiate(cfg.model)

/home/jovyan/data/kamila/detrex/detrex/config/config.py in get_config(config_path)
     50     if not os.path.exists(cfg_file):
     51         raise RuntimeError("{} not available in detrex configs!".format(config_path))
---> 52     cfg = LazyConfig.load(cfg_file)
     53     return cfg

/home/jovyan/data/kamila/detrex/detectron2/detectron2/config/lazy.py in load(filename, keys)
    209                 # 1. make filename appears in stacktrace
    210                 # 2. make load_rel able to find its parent's (possibly remote) location
--> 211                 exec(compile(content, filename, "exec"), module_namespace)
    212 
    213             ret = module_namespace

/home/jovyan/data/kamila/detrex/detrex/config/configs/common/data/dino_swin_large_384_4scale_12ep_graffiti.py in <module>
      1 import datetime
      2 from detrex.config import get_config
----> 3 from .models.dino_swin_large_384 import model
      4 # from detrex.projects.dino.configs.models.dino_swin_large_384 import model
      5 

/home/jovyan/data/kamila/detrex/detectron2/detectron2/config/lazy.py in new_import(name, globals, locals, fromlist, level)
    138             and (globals.get("__package__", "") or "").startswith(_CFG_PACKAGE_NAME)
    139         ):
--> 140             cur_file = find_relative_file(globals["__file__"], name, level)
    141             _validate_py_syntax(cur_file)
    142             spec = importlib.machinery.ModuleSpec(

/home/jovyan/data/kamila/detrex/detectron2/detectron2/config/lazy.py in find_relative_file(original_file, relative_import_path, level)
    125             cur_file += ".py"
    126         if not PathManager.isfile(cur_file):
--> 127             raise ImportError(
    128                 f"Cannot import name {relative_import_path} from "
    129                 f"{original_file}: {cur_file} has to exist."

ImportError: Cannot import name models.dino_swin_large_384 from /home/jovyan/data/kamila/detrex/detrex/config/configs/common/data/dino_swin_large_384_4scale_12ep_graffiti.py: /home/jovyan/data/kamila/detrex/detrex/config/configs/common/data/models/dino_swin_large_384.py has to exist.

I fixed it when I changed from .models.dino_swin_large_384 import model to from projects.dino.configs.models.dino_swin_large_384 import model

Yes, we use absolute import path here because we want to use a unified train_net for all the models

[rentainhe]

You can use the demo to inference your custom image or video using the pretrained weights. And more info can be found in the documentation: Inference demo with the pretrained weights.

As there is no more activity, I am closing the issue~ Feel free to reopen it if necessary. Or you can leave a new issue if you meet some other problems. @alrightkami