[AIC-py] hf image2text parser #821

jonathanlastmileai · 2024-01-09T19:55:49Z

[AIC-py] hf image2text parser

test

patch #816

python extensions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/run_hf_example.py extensions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/hf_local_example.aiconfig.json

-> "red fox in the woods"

extensions/HuggingFace/python/src/aiconfig_extension_hugging_face/__init__.py

rossdanlm · 2024-01-09T21:01:43Z

extensions/HuggingFace/python/src/aiconfig_extension_hugging_face/__init__.py


 LOCAL_INFERENCE_CLASSES = [
    "HuggingFaceText2ImageDiffusor",
    "HuggingFaceTextGenerationTransformer",
    "HuggingFaceTextSummarizationTransformer",
    "HuggingFaceTextTranslationTransformer",
    "HuggingFaceText2SpeechTransformer",
+    "HuggingFaceAutomaticSpeechRecognition",


cc @Ankush-lastmile you may have merge conflicts with your other PR?

nit: Can we also do these in alphabetical order?

Fixed in #862

rossdanlm · 2024-01-09T21:15:50Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+        Returns:
+            str: Serialized representation of the prompt and inference settings.
+        """
+        await ai_config.callback_manager.run_callbacks(


Can you add a TODO linking to #822 to fix later(and add automated testing. I'll do this later

what's broken?

We're not using the correct model_id so I need to pass this in so we can re-create the prompt

Added TODO comment in code in #862

rossdanlm · 2024-01-09T21:17:04Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+        inputs = validate_and_retrieve_image_from_attachments(prompt)
+
+        completion_params["inputs"] = inputs


cc @Ankush-lastmile when this lands, can you link to the Attachment format/standardizing inputs issue we mentioned? Jonathan you don't need to do any work, just making sure Ankush is aware

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

rossdanlm · 2024-01-09T21:20:06Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+        completion_data = await self.deserialize(prompt, aiconfig, parameters)
+        print(f"{completion_data=}")
+        inputs = completion_data.pop("inputs")
+        model = completion_data.pop("model")


This is never used again, why would we have it in completion data in the first place?

If it's never used, pls prefix with _model

it has to be removed from completion data. Something is definitely off here, I just don't know exactly what. cc @Ankush-lastmile

Fixed in #855 (I just deleted, we must've had model in the settings param). cc @saqadri who is working on adding this explicitly

rossdanlm · 2024-01-09T21:21:09Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+        model = completion_data.pop("model")
+        response = captioner(inputs, **completion_data)
+
+        output = ExecuteResult(output_type="execute_result", data=response, metadata={})


Oh sweet, so response is just purely text? nice! Also let's add "execution_count=0"

Fixed in #855

rossdanlm · 2024-01-09T21:22:19Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+        return prompt.outputs
+
+    def get_output_text(self, response: dict[str, Any]) -> str:
+        raise NotImplementedError("get_output_text is not implemented for HuggingFaceImage2TextTransformer")


Pls update to match others like the ones in text_generation.py

Fixed in #855

rossdanlm · 2024-01-09T21:22:43Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+
+def validate_attachment_type_is_image(attachment: Attachment):
+    if not hasattr(attachment, "mime_type"):
+        raise ValueError(f"Attachment has no mime type. Specify the image mimetype in the aiconfig")


Nit; add the work "Please" before "Specify"

Updated in #862

rossdanlm · 2024-01-09T21:24:46Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+            # See todo above, but for now only support uri's
+            raise ValueError(f"Attachment #{i} data is not a uri. Please specify a uri for the image attachment in prompt {prompt.name}.")


Doesn't have to be this diff, but please add support for base64 as well. This is important since if we want to chain prompts, some of our models output in base64 format (ex: text_2_image)

At very least, create an issue to track

Fixed in #856

rossdanlm · 2024-01-09T21:26:04Z

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

+        print(f"{completion_data=}")
+        inputs = completion_data.pop("inputs")
+        model = completion_data.pop("model")
+        response = captioner(inputs, **completion_data)


Does pipeline only support inputs as URI, or does it also work with base64 encoded? If not, pls make task that we need to convert from base64 --> image URI first

Fixed in #856

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py

test patch #816 ![pic](https://github.com/lastmile-ai/aiconfig/assets/148090348/d5cc26b3-6cb7-4331-af8a-92fd8c4e2471) python extensions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/run_hf_example.py extensions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/hf_local_example.aiconfig.json -> "red fox in the woods"

rossdanlm

Accepting to unblock, we'll do fast-follows in #835

Small fixes from comments from Sarmad + me from these diffs: - #854 - #855 - #821 Main things I did - rename `refine_chat_completion_params` --> `chat_completion_params` - edit `get_text_output` to not check for `OutputDataWithValue` - sorted the init file to be alphabetical - fixed some typos/print statements - made some error messages a bit more intuitive with prompt name - sorted some imports - fixed old class name `HuggingFaceAutomaticSpeechRecognition` --> `HuggingFaceAutomaticSpeechRecognitionTransformer` ## Test Plan These are all small nits and shouldn't change functionality

HF transformers: Small fixes nits Small fixes from comments from Sarmad + me from these diffs: - #854 - #855 - #821 Main things I did - rename `refine_chat_completion_params` --> `chat_completion_params` - edit `get_text_output` to not check for `OutputDataWithValue` - sorted the init file to be alphabetical - fixed some typos/print statements - made some error messages a bit more intuitive with prompt name - sorted some imports - fixed old class name `HuggingFaceAutomaticSpeechRecognition` --> `HuggingFaceAutomaticSpeechRecognitionTransformer` ## Test Plan These are all small nits and shouldn't change functionality

jonathanlastmileai mentioned this pull request Jan 9, 2024

[AIC-py] hf example #816

Draft

jonathanlastmileai force-pushed the pr821 branch 2 times, most recently from 9c946f9 to 4e7045b Compare January 9, 2024 20:38

jonathanlastmileai marked this pull request as ready for review January 9, 2024 20:39

jonathanlastmileai requested review from saqadri, rholinshead, suyoglastmileai, Ankush-lastmile and rossdanlm as code owners January 9, 2024 20:39

jonathanlastmileai force-pushed the pr821 branch from 4e7045b to 65180f4 Compare January 9, 2024 20:45

rossdanlm reviewed Jan 9, 2024

View reviewed changes

extensions/HuggingFace/python/src/aiconfig_extension_hugging_face/__init__.py Show resolved Hide resolved

rossdanlm reviewed Jan 9, 2024

View reviewed changes

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py Outdated Show resolved Hide resolved

rossdanlm reviewed Jan 9, 2024

View reviewed changes

...sions/HuggingFace/python/src/aiconfig_extension_hugging_face/local_inference/image_2_text.py Show resolved Hide resolved

jonathanlastmileai force-pushed the pr821 branch from 65180f4 to 448d52a Compare January 9, 2024 21:40

rossdanlm mentioned this pull request Jan 9, 2024

Fast Follows for image2Text HF model parser #835

Closed

rossdanlm approved these changes Jan 9, 2024

View reviewed changes

rossdanlm merged commit 3f0cbce into main Jan 9, 2024

rossdanlm deleted the pr821 branch January 9, 2024 23:58

rossdanlm mentioned this pull request Jan 10, 2024

HF transformers: Small fixes nits #862

Merged

rossdanlm mentioned this pull request Jan 11, 2024

[docs] updated getting started #516

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIC-py] hf image2text parser #821

[AIC-py] hf image2text parser #821

jonathanlastmileai commented Jan 9, 2024 •

edited

Loading

rossdanlm Jan 9, 2024

rossdanlm Jan 9, 2024

rossdanlm Jan 10, 2024

rossdanlm Jan 9, 2024

jonathanlastmileai Jan 9, 2024

rossdanlm Jan 9, 2024

rossdanlm Jan 10, 2024

rossdanlm Jan 9, 2024

rossdanlm Jan 9, 2024 •

edited

Loading

jonathanlastmileai Jan 9, 2024

rossdanlm Jan 10, 2024

rossdanlm Jan 9, 2024 •

edited

Loading

rossdanlm Jan 10, 2024

rossdanlm Jan 9, 2024

rossdanlm Jan 10, 2024

rossdanlm Jan 9, 2024

rossdanlm Jan 10, 2024

rossdanlm Jan 9, 2024

rossdanlm Jan 10, 2024

rossdanlm Jan 9, 2024

rossdanlm Jan 10, 2024

rossdanlm left a comment

		inputs = validate_and_retrieve_image_from_attachments(prompt)

		completion_params["inputs"] = inputs

		# See todo above, but for now only support uri's
		raise ValueError(f"Attachment #{i} data is not a uri. Please specify a uri for the image attachment in prompt {prompt.name}.")

[AIC-py] hf image2text parser #821

[AIC-py] hf image2text parser #821

Conversation

jonathanlastmileai commented Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossdanlm Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossdanlm Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossdanlm left a comment

Choose a reason for hiding this comment

jonathanlastmileai commented Jan 9, 2024 •

edited

Loading

rossdanlm Jan 9, 2024 •

edited

Loading

rossdanlm Jan 9, 2024 •

edited

Loading