Add converter from Pascal VOC to Weed COCO #49

jnothman · 2020-09-01T06:36:56Z

~~A first draft. Not fully tested.~~

Fixes #33

hlydecker · 2020-09-02T08:48:12Z

conversion_tools/voc_to_json/voc_to_json.py

+def voc_to_coco(
+    xml_dir: Path, image_dir: Path, category_mapping: Optional[Mapping[str, int]] = None
+):
+    """Convert VOC to MS COCO images and annotations


Nice documentation; would be good to update the existing ones to be similar.

hlydecker · 2020-09-02T08:52:27Z

conversion_tools/voc_to_json/voc_to_json.py

+        obj = json.load(path)
+
+
+def main():


I've looked this up because I didn't understand what the heck the main function is. Looks like I have some reading to do; this seems like something I need to adopt to help me fully transition to writing scripts vs interactively using notebooks.

It basically serves two purposes:

allows the user to use this file both as a library and as a script

encapsulates the scope to avoid errors

hlydecker · 2020-09-02T08:53:33Z

conversion_tools/voc_to_json/voc_to_json.py

+        )
+
+    if use_default_category_mapping:
+        # TODO: define role somehow??


For now role will have to be hard coded. Perhaps if we have a filled out agcontext yaml we can use that to populate the category field as well?

Yeah, needs some way to be specified. I wonder whether "name" should include species and role somehow...

jnothman · 2020-09-03T01:52:33Z

I can now run the following:

python conversion_tools/voc_to_json/voc_to_json.py --voc-dir /Volumes/research-data/PRJ-iweeds/Narrabri/artificial_illumination/wheat/20190728_Z16/VOC --image-dir /Volumes/research-data/PRJ-iweeds/Narrabri/artificial_illumination/wheat/20190728_Z16/images --agcontext /Volumes/research-data/PRJ-iweeds/agcontext-junk-example.yaml  -o /tmp/coco_from_voc-test.json
cat /tmp/coco_from_voc-test.json | python conversion_tools/draft\ export\ to\ elastic.py --thumbnail-dir artificial | curl -X POST localhost:9200/_bulk  -H 'Content-Type: application/json' --data-binary @-

hlydecker · 2020-09-04T05:06:49Z

I can now run the following:

python conversion_tools/voc_to_json/voc_to_json.py --voc-dir /Volumes/research-data/PRJ-iweeds/Narrabri/artificial_illumination/wheat/20190728_Z16/VOC --image-dir /Volumes/research-data/PRJ-iweeds/Narrabri/artificial_illumination/wheat/20190728_Z16/images --agcontext /Volumes/research-data/PRJ-iweeds/agcontext-junk-example.yaml  -o /tmp/coco_from_voc-test.json
cat /tmp/coco_from_voc-test.json | python conversion_tools/draft\ export\ to\ elastic.py --thumbnail-dir artificial | curl -X POST localhost:9200/_bulk  -H 'Content-Type: application/json' --data-binary @-

I like this; it seems like a pretty clean and relatively straightforward process. It helps me imagine more of what our pipeline for user provided data might look like.

hlydecker · 2020-09-23T23:32:07Z

weedcoco/draft export to elastic.py

@@ -31,6 +31,12 @@
        id_lookup[key, obj["id"]] = obj


+for agcontext in coco["agcontexts"]:
+    # massage for ES
+    if agcontext.get("camera_fov") == "variable":


Unfortunately it looks like we will need to stick with this "massaging" to make this work. Elasticsearch will not allow a field to have multiple potential types, and I think it would be a bit silly to make this into some sort of nested field array where we have something like:

camera_fov: { is_fixed: False fov: null }

I'm okay with this massaging, as long as any slider over FOV that we have also allows us to retrieve NULLs.

hlydecker · 2020-09-23T23:34:55Z

weedcoco/importers/voc.py

+            "id": annotation_id,
+            "image_id": image_id,
+            "category_id": category_mapping[voc_object.find("name").text],
+            # TODO: handle truncated, difficult?


What do you mean by truncated annotation?

VOC allows an annotation to be tagged as "truncated". I think that means "only part of the object being labelled is visible in the image; it is cut off by the edge of the picture". I don't know what "difficult" is suppose to mean.

From http://host.robots.ox.ac.uk/pascal/VOC/voc2006/htmldoc/index.html

Huh I wonder if people would consider our rye grass "difficult"...

I think "difficult" here means "so difficult it's not worth demeriting a system for failing it"

jnothman

I've not quite finished this PR, btw. I'll mark it as draft. I felt #82 and #84 would support it well.

jnothman · 2020-09-23T23:51:52Z

weedcoco/draft export to elastic.py

@@ -31,6 +31,12 @@
        id_lookup[key, obj["id"]] = obj


+for agcontext in coco["agcontexts"]:
+    # massage for ES
+    if agcontext.get("camera_fov") == "variable":


I'm okay with this massaging, as long as any slider over FOV that we have also allows us to retrieve NULLs.

jnothman · 2020-09-23T23:52:59Z

weedcoco/importers/voc.py

+            "id": annotation_id,
+            "image_id": image_id,
+            "category_id": category_mapping[voc_object.find("name").text],
+            # TODO: handle truncated, difficult?


VOC allows an annotation to be tagged as "truncated". I think that means "only part of the object being labelled is visible in the image; it is cut off by the edge of the picture". I don't know what "difficult" is suppose to mean.

jnothman · 2020-09-23T23:53:49Z

weedcoco/importers/voc.py

+            "id": annotation_id,
+            "image_id": image_id,
+            "category_id": category_mapping[voc_object.find("name").text],
+            # TODO: handle truncated, difficult?


From http://host.robots.ox.ac.uk/pascal/VOC/voc2006/htmldoc/index.html

jnothman · 2020-09-24T06:07:43Z

Writing well-tested code from scratch takes quite a lot of work! This is now ready for review.

jnothman · 2020-09-24T06:16:21Z

Oh no! CI is using a recent beta version of black, which differs on how it formats weedcoco.utils!

jnothman · 2020-10-08T00:22:02Z

.github/workflows/pytest.yml

@@ -26,10 +26,6 @@ jobs:
      - uses: actions/checkout@v2
      - name: Install
        run: |
-          pip install . pytest pytest-cov


I don't think these changes should be undone by the present branch. Failed merge?

jnothman · 2020-10-08T00:23:26Z

.github/workflows/validate_json.yml

          cd ../deepweeds_to_json
          curl -L https://github.com/AlexOlsen/DeepWeeds/archive/51e3fab.tar.gz | tar xzv
          mv DeepWeeds-51e3fab*/* .
          # fake some images: make them all be a pic from cwfid
          mkdir deepweeds_images_full
          for fname in $(cut -d, -f1 < labels.csv); do ln -s ../../cwfid_to_json/cwfid_images/001_image.png deepweeds_images_full/$fname; done
          python -m weedcoco.importers.deepweeds --labels-dir labels --image-dir deepweeds_images_full -o deepweeds_imageinfo.json
-      - name: Validate WeedCOCO


This also shouldn't have been removed in merge.

codecov · 2020-10-08T00:23:56Z

Codecov Report

Merging #49 into master will decrease coverage by 19.39%.
The diff coverage is 31.42%.

@@             Coverage Diff             @@
##           master      #49       +/-   ##
===========================================
- Coverage   31.48%   12.08%   -19.40%     
===========================================
  Files           6        7        +1     
  Lines         343      364       +21     
===========================================
- Hits          108       44       -64     
- Misses        235      320       +85

Flag	Coverage Δ
#weedcoco	`12.08% <31.42%> (-19.40%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
weedcoco/importers/cwfid.py	`0.00% <0.00%> (ø)`
weedcoco/importers/deepweeds.py	`0.00% <ø> (ø)`
weedcoco/importers/voc.py	`0.00% <0.00%> (ø)`
weedcoco/utils.py	`0.00% <0.00%> (ø)`
weedcoco/validation.py	`0.00% <0.00%> (-80.83%)`	⬇️
weedcoco/tests/importers/test_voc.py	`97.77% <97.77%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 113df9e...b6cb484. Read the comment docs.

jnothman · 2020-10-08T00:29:21Z

I'll reopen this. Trying to revert the merge, I pushed the wrong parent, which closed this PR irreversibly, it seems.

hlydecker reviewed Sep 2, 2020

View reviewed changes

jnothman mentioned this pull request Sep 3, 2020

Will need an explicit schema on ElasticSearch; or change "variable" and "na" into NULL #61

Closed

jnothman mentioned this pull request Sep 18, 2020

Make weedcoco into a Python package #80

Merged

hlydecker reviewed Sep 23, 2020

View reviewed changes

jnothman commented Sep 23, 2020

View reviewed changes

jnothman mentioned this pull request Sep 24, 2020

When indexing, remove 'variable' where it creates type conflicts #89

Merged

jnothman added the import label Sep 24, 2020

jnothman commented Oct 8, 2020

View reviewed changes

jnothman closed this Oct 8, 2020

jnothman force-pushed the voc-convert branch from b6cb484 to 113df9e Compare October 8, 2020 00:25

jnothman mentioned this pull request Oct 8, 2020

Add converter from Pascal VOC to Weed COCO #101

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add converter from Pascal VOC to Weed COCO #49

Add converter from Pascal VOC to Weed COCO #49

jnothman commented Sep 1, 2020 •

edited

hlydecker Sep 2, 2020

hlydecker Sep 2, 2020

jnothman Sep 2, 2020

hlydecker Sep 2, 2020

jnothman Sep 3, 2020

jnothman commented Sep 3, 2020 •

edited

hlydecker commented Sep 4, 2020

hlydecker Sep 23, 2020

jnothman Sep 23, 2020

hlydecker Sep 23, 2020

jnothman Sep 23, 2020

jnothman Sep 23, 2020

hlydecker Sep 24, 2020

jnothman Sep 24, 2020

jnothman left a comment

jnothman Sep 23, 2020

jnothman Sep 23, 2020

jnothman Sep 23, 2020

jnothman commented Sep 24, 2020

jnothman commented Sep 24, 2020

jnothman Oct 8, 2020

jnothman Oct 8, 2020

codecov bot commented Oct 8, 2020

jnothman commented Oct 8, 2020

Add converter from Pascal VOC to Weed COCO #49

Add converter from Pascal VOC to Weed COCO #49

Conversation

jnothman commented Sep 1, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnothman commented Sep 3, 2020 • edited

hlydecker commented Sep 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnothman commented Sep 24, 2020

jnothman commented Sep 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 8, 2020

Codecov Report

jnothman commented Oct 8, 2020

jnothman commented Sep 1, 2020 •

edited

jnothman commented Sep 3, 2020 •

edited