Skip to content

Add Falcon Perception model support to inference-models#2213

Draft
hansent wants to merge 1 commit intomainfrom
falcon-perception
Draft

Add Falcon Perception model support to inference-models#2213
hansent wants to merge 1 commit intomainfrom
falcon-perception

Conversation

@hansent
Copy link
Copy Markdown
Collaborator

@hansent hansent commented Apr 8, 2026

Integrate TII's Falcon Perception (600M param dense Transformer) for open-vocabulary object detection and instance segmentation from natural language text prompts. Wraps the falcon-perception==1.0.0 PyPI package using its BatchInferenceEngine for the simplest inference path.

  • Two model classes: detection-only (OpenVocabularyObjectDetectionModel) and segmentation (InstanceSegmentationModel), both torch-only
  • Registered in models_registry as "falcon-perception" architecture
  • falcon-perception added as optional extra (conflicts with JP6/older CUDA extras due to torch>=2.11.0 + numpy>=2 requirements)
  • Configuration constants for mask threshold, max tokens, image sizes
  • 20 unit tests for post-processing helpers (bbox pairing, coordinate conversion, image format handling, edge cases)

What does this PR do?

Related Issue(s):

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Other:

Testing

  • I have tested this change locally
  • I have added/updated tests for this change

Test details:

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in hard-to-understand areas
  • My changes generate no new warnings or errors
  • I have updated the documentation accordingly (if applicable)

Additional Context

Integrate TII's Falcon Perception (600M param dense Transformer) for
open-vocabulary object detection and instance segmentation from natural
language text prompts. Wraps the falcon-perception==1.0.0 PyPI package
using its BatchInferenceEngine for the simplest inference path.

- Two model classes: detection-only (OpenVocabularyObjectDetectionModel)
  and segmentation (InstanceSegmentationModel), both torch-only
- Registered in models_registry as "falcon-perception" architecture
- falcon-perception added as optional extra (conflicts with JP6/older
  CUDA extras due to torch>=2.11.0 + numpy>=2 requirements)
- Configuration constants for mask threshold, max tokens, image sizes
- 20 unit tests for post-processing helpers (bbox pairing, coordinate
  conversion, image format handling, edge cases)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant