Skip to content

DEVX-583 Added multimodal loader support in pipeline#11

Merged
mogith-pn merged 14 commits intomainfrom
DEVX-583-multimodal
Sep 26, 2024
Merged

DEVX-583 Added multimodal loader support in pipeline#11
mogith-pn merged 14 commits intomainfrom
DEVX-583-multimodal

Conversation

@mogith-pn
Copy link
Copy Markdown
Contributor

@mogith-pn mogith-pn commented Aug 29, 2024

Feature - Multimodal loader support for PDFs

Added multimodal loader support for PDFs, which retains the same structure as text pipeline but modified the loaders and base class to handle multimodal data type which reuses the existing 'visualclassification', 'textclassification' loaders.

Sample structure of multimodal pipeline -

  • clarifai_datautils/
    • multimodal/
      • pipeline/
        • base.py
        • extractors.py
        • cleaners.py
        • loaders.py
        • PDF.py
        • README.md
        • Text.py
      • init.py

Copy link
Copy Markdown
Contributor

@sanjaychelliah sanjaychelliah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Work! Added some comments

Comment thread clarifai_datautils/constants/base.py Outdated
Comment thread setup.py Outdated
Comment thread clarifai_datautils/multimodal/pipeline/PDF.py
Comment thread clarifai_datautils/multimodal/pipeline/PDF.py Outdated
Comment thread clarifai_datautils/multimodal/pipeline/loaders.py Outdated
Comment thread clarifai_datautils/multimodal/pipeline/PDF.py Outdated
Copy link
Copy Markdown
Contributor

@sanjaychelliah sanjaychelliah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Work! Added some comments

Comment thread clarifai_datautils/multimodal/pipeline/base.py Outdated
Comment thread tests/pipelines/test_multimodal_pipelines.py
Comment thread clarifai_datautils/multimodal/pipeline/base.py Outdated
Comment thread clarifai_datautils/multimodal/pipeline/base.py Outdated
sanjaychelliah and others added 3 commits September 26, 2024 12:52
Co-authored-by: sanjaychelliah <65780631+sanjaychelliah@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

@sanjaychelliah sanjaychelliah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mogith-pn mogith-pn merged commit a042320 into main Sep 26, 2024
This was referenced Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants