Prevent folder name conflicts in label() call #93
Conversation
|
NB: This PR fixes #53. |
| if output_folder is None: | ||
| output_folder = input_folder + "_labeled" | ||
|
|
||
| os.makedirs(output_folder, exist_ok=True) |
There was a problem hiding this comment.
Logic should be:
- If
output_folderalready exists - Check whether it was created with the same config (same base model, same ontology)
- If the config is the same, continue
- If the config is different, move the old folder to a backup location (
{output_folder}-{old_timestamp})
To do this we need to store a hash of the ontology in the config file.
We should also rename this config.json to be .autodistill.json so it's not conflicting with other stuff & is clear where it came from/what it's used for.
There was a problem hiding this comment.
To do the hashing we'll add a .hash() method to the Ontology base class that's just an md5 of the JSON of the ontology. Subclasses can override this to create their own definition of what it means for an ontology to be "different" from another.
|
|
||
| if not os.path.exists(annotation_path): | ||
| detections = self.predict(f_path) | ||
| detections_map[f_path_short] = detections |
There was a problem hiding this comment.
Save these to disk as we go vs to memory so that if you cancel or crash in the middle you don't lose all your progress
|
|
||
| images_map = {} | ||
| detections_map = {} | ||
|
|
||
| # if output_folder/autodistill.json exists | ||
| if os.path.exists(output_folder + "/data.yaml"): | ||
| dataset = sv.DetectionDataset.from_yolo( |
There was a problem hiding this comment.
Don't load all the images into memory here in case the dataset is huge; we just need the list of filenames
| files = glob.glob(input_folder + "/*" + extension) | ||
| progress_bar = tqdm(files, desc="Labeling images") | ||
| # iterate through images in input_folder | ||
|
|
There was a problem hiding this comment.
Save the data.yaml file first instead of at the end so it's there next run
| config["roboflow_tags"] = roboflow_tags | ||
| config["task"] = "detection" | ||
|
|
||
| with open(os.path.join(output_folder, "config.json"), "w+") as f: |
This PR appends a timestamp of when
label()was called to the output folder label name if the provided folder name already exists and contains images.This PR prevents the scenario where
label()labels a dataset (which could be hundreds or thousands of images) then returns an error after labeling because the existing folder already contains an image with the same name as one in the newly-labeled dataset.