Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Extraction #575

Merged
merged 25 commits into from Jan 3, 2019
Merged

Feature Extraction #575

merged 25 commits into from Jan 3, 2019

Conversation

@jamesmcclain
Copy link
Member

@jamesmcclain jamesmcclain commented Nov 26, 2018

Overview

Adds ability to produce a GeoJSON file containing semantic segmentation predictions.

The ProtoBuf schema has been updated to include a list of vector outputs.

The segmentation integration test gives the following results:

mask
screenshot from 2018-12-05 14-38-44

Evaluation of vector outputs has been left for future work (see #611 ).

polygon output mode left for future work (see #612 ).

Checklist

  • Ran scripts/format_code and commited any changes
  • Documentation updated if needed
  • PR has a name that won't get you publicly shamed for vagueness

Notes

Optional. Ancillary topics, caveats, alternative strategies that didn't work out, anything else.

Testing Instructions

  • How to test this PR
  • Prefer bulleted description
  • Start after checking out this branch
  • Include any setup required, such as rebuilding the Docker image.
  • Include test case, and expected output if not captured by automated tests.

Closes #544
Closes #611
Closes #612
Closes #564

@jamesmcclain jamesmcclain changed the title [WiP] Feature extraction [WiP] Feature Extraction Nov 27, 2018
@jamesmcclain jamesmcclain force-pushed the feature-extraction branch 4 times, most recently from 67aa06a to e704aa2 Nov 28, 2018
@jamesmcclain jamesmcclain changed the title [WiP] Feature Extraction Feature Extraction Nov 28, 2018
@jamesmcclain jamesmcclain force-pushed the feature-extraction branch from ac65ed8 to a3afcb8 Nov 29, 2018
@lewfish
Copy link
Contributor

@lewfish lewfish commented Nov 29, 2018

I haven't looked at this in any detail yet, but one request is to add polygon mode which extracts polygons without doing the extra right-angle stuff for buildings.

requirements-dev.txt Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
rastervision/data/scene_config.py Outdated Show resolved Hide resolved
rastervision/data/scene_config.py Outdated Show resolved Hide resolved
@jamesmcclain jamesmcclain deleted the feature-extraction branch Dec 3, 2018
@jamesmcclain jamesmcclain restored the feature-extraction branch Dec 3, 2018
@jamesmcclain jamesmcclain reopened this Dec 3, 2018
@jamesmcclain jamesmcclain force-pushed the feature-extraction branch 2 times, most recently from 70afc8e to a012077 Dec 5, 2018
@lewfish
Copy link
Contributor

@lewfish lewfish commented Dec 6, 2018

Evaluation of vector outputs has been left for future work.

polygon output mode left for future work.

Whenever you decide that something is outside the scope of a PR and is left for future work, it is a good idea to make an issue for those tasks and link to them in the PR.

Also, reminder to remove unused boilerplate text from the PR description.

Copy link
Contributor

@lewfish lewfish left a comment

Will test after change.

@jamesmcclain jamesmcclain force-pushed the feature-extraction branch 2 times, most recently from 2b9e790 to 86a0d2e Dec 12, 2018
@lewfish
Copy link
Contributor

@lewfish lewfish commented Dec 13, 2018

I tested this using a modification of the Vegas example in
azavea/raster-vision-examples@65417f6 azavea/raster-vision-examples@337b244

The command run was

export ROOT_URI=/opt/data/lf-dev/rv-output/vegas-spacenet
rastervision run local -e spacenet.vegas \
    -a test True \
    -a use_remote_data False \
    -a root_uri ${ROOT_URI} \
    -a target buildings \
    -a task_type semantic_segmentation

If you don't have the Vegas data on your local filesystem you can switch use_remote_data to True.

I noticed that there were some junk predictions of buildings and expected there to be some polygons output for them, but there weren't any.

@lewfish
Copy link
Contributor

@lewfish lewfish commented Dec 13, 2018

Note that I updated the branch/commit that the Vegas test is in above.

@jamesmcclain jamesmcclain force-pushed the feature-extraction branch from 86a0d2e to b68b990 Dec 18, 2018
@jamesmcclain
Copy link
Member Author

@jamesmcclain jamesmcclain commented Dec 21, 2018

With jamesmcclain#1

Ground truth

screenshot_2018-12-21_10-24-40

Predictions

screenshot_2018-12-21_10-24-30

Eval

The file /tmp/XXX/eval/buildings_semantic_segmentation/eval.json should look something like this:

[
    {
        "class_id": 1,
        "count_error": 399941.64566854003,
        "precision": 0.03893706315187772,
        "class_name": "Building",
        "f1": 0.06278322228817987,
        "gt_count": 303916,
        "recall": 0.16918819673857252
    },
    {
        "class_id": 2,
        "count_error": 401015.0285372315,
        "precision": 0.12290437362221147,
        "class_name": "Background",
        "f1": 0.04184535492697485,
        "gt_count": 1386084,
        "recall": 0.02546238178927107
    },
    {
        "class_id": -2,
        "count_error": 13.26605504587156,
        "precision": 0.9267246514952938,
        "class_name": "vector-polygons",
        "f1": 0.7137699626231737,
        "gt_count": 109,
        "recall": 0.5871559633027523
    },
    {
        "class_id": null,
        "count_error": 400796.1506896892,
        "precision": 0.10785718274954362,
        "class_name": "average",
        "f1": 0.04565374401855821,
        "gt_count": 1690109,
        "recall": 0.051343434062536795
    }
]

@jamesmcclain jamesmcclain force-pushed the feature-extraction branch from ca93b5d to 81e2cb8 Jan 3, 2019
@jamesmcclain jamesmcclain merged commit 7eafab6 into azavea:develop Jan 3, 2019
1 check passed
@jamesmcclain jamesmcclain deleted the feature-extraction branch Jan 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants