feat(search): add Tika object detection captions and labels#12072
Closed
paul43210 wants to merge 17 commits intoowncloud:masterfrom
Closed
feat(search): add Tika object detection captions and labels#12072paul43210 wants to merge 17 commits intoowncloud:masterfrom
paul43210 wants to merge 17 commits intoowncloud:masterfrom
Conversation
This change enables photo gallery applications to access EXIF metadata through the search API. Changes: - Add KQL query support for photo fields: - photo.takenDateTime (datetime range queries) - photo.cameraMake, photo.cameraModel (keyword queries) - photo.fNumber, photo.focalLength, photo.iso (numeric queries) - Configure Bleve index to store photo fields (Store=true) - Map photo metadata from Bleve hits to Entity.Photo protobuf - Add oc:photo-* properties to WebDAV search responses - Add unit tests for photo field queries The photo metadata is extracted by Tika during indexing and stored in the Bleve search index. This enables queries like: photo.takenDateTime:[2023-01-01 TO 2023-12-31] photo.cameraMake:Canon Signed-off-by: Paul Faure <paul@faure.ca> Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add location field mappings with Store=true in Bleve index - Rename properties to oc:photo-location-* for consistency - Add altitude support alongside latitude/longitude - Requires index rebuild to take effect
- Add exposure numerator/denominator to search results - Add altitude extraction from Tika GPS metadata - All photo EXIF fields now fully exposed via WebDAV search
Extract photo and location property building into separate helper functions (appendPhotoProps, appendLocationProps) to reduce cognitive complexity from 41 to under 15. This addresses SonarCloud issue go:S3776.
Extract repetitive nil/zero checks into reusable helper functions: - appendStringProp: handles *string with nil and empty checks - appendFloat32Prop: handles *float32 with nil and zero checks - appendFloat64Prop: handles *float64 with nil check - appendInt32Prop: handles *int32 with nil and zero checks This reduces appendPhotoProps complexity from 18 to under 15 and appendLocationProps complexity as well. Addresses SonarCloud issue go:S3776.
The context parameter was not being used in the function. Also removed from multistatusResponse and removed unused context import. Addresses SonarCloud issue go:S1172.
- Add comment explaining why appendFloat64Prop doesn't check for zero (GPS coordinates 0.0 are valid for equator/prime meridian) - Fix changelog to list only implemented fields (removed exposureTime, exposureBias, flash, meteringMode, whiteBalance; added exposureNumerator, exposureDenominator)
Extract OBJECT/CAPTION metadata from Tika's ObjectRecognitionParser, index them in Bleve, make them queryable via KQL (objectLabel:/objectCaption:), and expose them as oc:object-labels/oc:object-captions WebDAV properties. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…r for captions Tika's Show and Tell / Inception V3 models produce log-probability confidence scores in the 0.0001–0.002 range, far below the previous 0.10 threshold which silently discarded all results. Lower threshold to 0.0001 for both labels and captions. Switch ObjectCaptions Bleve analyzer from lowercaseKeyword to fulltext so that word-level search (e.g. objectCaption:dog) works on multi-word caption strings. Add test coverage for low-confidence extraction and threshold filtering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
7 tasks
…awDesc The ObjectLabels and ObjectCaptions fields were added to the Entity Go struct but the protobuf rawDesc (binary file descriptor) was not regenerated. protobuf-go v2 uses rawDesc for serialization, so these fields were silently dropped during gRPC communication between the search and webdav services — causing search to find matching documents but return empty metadata in the WebDAV XML response. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…oval (TIKA-4500) ObjectCaptions and ObjectLabels fields are extractor-agnostic by design. Tika 3.x ObjectRecognitionParser is supported until June 2026. Tika 4.x removed advancedmedia module but fields remain available for any content extractor that provides caption/label metadata. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tika returns ISO as `exif:IsoSpeedRatings` (standard EXIF tag) but the code only checked `Base ISO` (Nikon-specific MakerNote tag). Check the standard key first, falling back to `Base ISO` for Nikon cameras. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
8 tasks
Loop over candidate metadata keys instead of duplicating the parse/set block in each branch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Author
|
Closing this PR. With Tika 4.x expected this summer (which removes the |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
oCIS can extract rich metadata from images via Apache Tika, including object detection labels (Inception V3) and image captions (Show and Tell model). However, this metadata was not being captured, indexed, or exposed through the search API — meaning users had no way to search for photos by their visual content (e.g., finding all photos containing dogs or beach scenes).
This builds on the photo EXIF metadata work in #11912, extending search capabilities from camera/technical metadata to AI-generated visual content descriptions.
Summary
OBJECTandCAPTIONmetadata from the configured content extractorobjectLabel:andobjectCaption:KQL field prefixes for querying (e.g.,objectCaption:dog)oc:object-labelsandoc:object-captionsWebDAV properties in search responsesobject_labelsandobject_captionsfields to the protobuf Entity messagerawDescto include fields 20/21, fixing silent data loss during gRPC serializationExtractor Compatibility Note
The
ObjectCaptionsandObjectLabelsindexed fields are populated by metadata from the configured content extractor via the standard extraction pipeline. These fields are extractor-agnostic by design.Current status:
ObjectRecognitionParserin thetika-parser-advancedmedia-package.The oCIS-side indexed fields support the broader ecosystem of content analysis tools. Any extractor that populates
CAPTIONor object label metadata will automatically have its results indexed and searchable through theObjectCaptionsandObjectLabelsfields. TheSEARCH_EXTRACTOR_TYPEconfiguration makes this pluggable.Test plan
objectCaption:dogreturns results for photos containing dogsobjectLabel:retrieverreturns results for photos with matching labelsNote
After deploying a binary with this change, the Bleve search index must be rebuilt for the caption analyzer change to take effect on existing documents. New documents indexed after deployment will use the fulltext analyzer automatically.
🤖 Generated with Claude Code