Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Florence 2 support #1296

Merged
merged 12 commits into from
Jun 21, 2024
Merged

Conversation

LinasKo
Copy link
Collaborator

@LinasKo LinasKo commented Jun 20, 2024

Description

This PR adds Florence 2 support for supervision.

Run it with transfomers and parse with from_lmm(sv.LMM.FLORENCE_2, result).

No extra documentation was added to from_lmm at this point.

Type of change

  • New feature (non-breaking change which adds functionality)

How has this change been tested, please provide a testcase or example of how you tested the change?

Usage and testing in Colab: https://colab.research.google.com/drive/19XoQZ6LRdTUysAHUp-3JsfT_67T0rffE#scrollTo=CwPzmI9wOSQ6

Any specific deployment considerations

Run the Colab in High-RAM mode.

Docs

  • Docs updated? What were the changes:

@LinasKo LinasKo requested a review from SkalskiP June 20, 2024 12:03
@LinasKo
Copy link
Collaborator Author

LinasKo commented Jun 20, 2024

I found there's some tasks I overlooked.

In addition to examples on their Hugging Face page, they have 5-6 more on their colab.

At the end of today I'm leaving this in a working-but-incomplete state:
I've added all tasks I could find, updated the Colab, but am not yet happy yet.

  • I know multiple places where the code can be simplified
  • REGION_TO_CATEGORY and REGION_TO_DESCRIPTION boxes are drawn incorrectly.
  • I haven't exhaustively tested the empty cases, or researched the possible outputs. In many cases, e.g. REFERRING_EXPRESSION_SEGMENTATION, I'm getting one large [[[polygon]]], regardless of what inputs I try.
  • It's late, so I simply ran out of time

Planning to dig deeper, simplify and add unit tests tomorrow.

@LinasKo
Copy link
Collaborator Author

LinasKo commented Jun 21, 2024

Hi @SkalskiP,

This now has:

  • Support for segmentation
  • Support for regions methods
  • Unit tests

Where it's lacking is:

  • While I looked into text prompt, I made no effort to check whether users can provide their class names into the transformers method. In a few cases, label lists of ["", "", "", ""] are returned. My implementation ignores those.
  • In some cases such as REFERRING_EXPRESSION_SEGMENTATION, I always get [[[polygon]]], regardless of what image and prompt I try. I did figure out what the layers represent and left some comments.
  • Instead of returning nothing, some methods return very narrow long polygons. We could do post-processing here.

I invite to look at the Colab and play around with the results.

@SkalskiP SkalskiP merged commit d5cebd4 into roboflow:develop Jun 21, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants