PR-500 simplified client wrapper functions by mogith-pn · Pull Request #562 · Clarifai/clarifai-python

mogith-pn · 2025-04-23T15:58:28Z

Why

https://clarifai.atlassian.net/browse/PR-500
The client wrapper function in model upload code, still needed some abstraction for converting images/video/audio into openai compatible format before sending to chat function.

How

Since it can be abstacted into the runner data utils function, moved those preprocessing steps into SDK.

Tests

PR tested with model upload into production after making changes - https://clarifai.com/mogith-p-n/test-co/models/smollm
Uploaded model code - https://github.com/Clarifai/model-uploads/tree/smollm-model/SmolLM/SmolLM2-1.7B-Instruct
Tested all the functions after deployment.

Copilot

Pull Request Overview

This PR simplifies the client wrapper functions by introducing a unified helper to format OpenAI chat messages with various media types (image, audio, video) and by adding base64 conversion methods to the corresponding data types.

Updated PIL image conversions by importing PILImage and aligning type hints.
Added build_openai_chat_format and internal helper functions to process images, audio, and video.
Extended data types with new to_base64_str methods for Image, Audio, and Video objects.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
clarifai/runners/utils/data_utils.py	Updated image conversion functions and added media processing functions for chat messages.
clarifai/runners/utils/data_types.py	Added to_base64_str utility methods for converting media bytes to base64 strings.

Comments suppressed due to low confidence (2)

clarifai/runners/utils/data_utils.py:132

[nitpick] Consider using a distinct local variable for the base64 string conversion in _process_audio instead of reassigning the parameter 'audio', to improve code clarity.

if audio.bytes:

clarifai/runners/utils/data_utils.py:158

[nitpick] Consider using a separate variable to hold the base64 string in _process_video rather than overwriting the parameter 'video', for improved readability.

if video.bytes:

luv-bansal

Looks good, but not sure , should we add build_openai_chat_format function to utils/openai_convertor.py file?

luv-bansal · 2025-05-15T07:39:15Z

  return True


+def build_openai_chat_format(prompt: str, image: Image, images: List[Image], audio: Audio,


Not sure, do you think we could move this to here

luv-bansal · 2025-05-22T07:15:42Z

  return True


+def build_openai_chat_format(prompt: str, image: Image, images: List[Image], audio: Audio,


also should be rename this function it to build_openai_messages or just openai_messages like we have openai_response function here https://github.com/Clarifai/clarifai-python/blob/master/clarifai/runners/utils/openai_convertor.py#L98?

Copilot

Pull Request Overview

This PR centralizes preprocessing of image, audio, and video inputs into SDK utilities and provides helpers to assemble OpenAI-compatible chat messages.

Introduce openai_convertor.py with build_openai_messages and is_openai_chat_format
Add process_image, process_audio, and process_video in data_utils.py
Extend data types with to_base64_str methods and update import in model_client.py

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
clarifai/runners/utils/openai_convertor.py	Added message builder and chat-format validator for multimodal inputs
clarifai/runners/utils/data_utils.py	Implemented processing helpers for Image, Audio, and Video
clarifai/runners/utils/data_types/data_types.py	Added `to_base64_str` to Image, Audio, and Video types
clarifai/client/model_client.py	Updated `is_openai_chat_format` import to the new converter

Comments suppressed due to low confidence (2)

clarifai/runners/utils/openai_convertor.py:1

[nitpick] The filename uses 'convertor' spelling; consider renaming to 'converter' for consistency with common terminology.

import time

clarifai/runners/utils/openai_convertor.py:173

Add unit tests for multimodal message construction and validation (process_image, process_audio, process_video, build_openai_messages, is_openai_chat_format) to ensure correct behavior across modalities.

def build_openai_messages(

Copilot · 2025-05-22T14:16:37Z

+    prompt: str,
+    image: Image,
+    images: List[Image],
+    audio: Audio,
+    audios: List[Audio],
+    video: Video,
+    videos: List[Video],
+    messages: List[Dict],


Consider making parameters optional with default values (e.g., prompt: str = '', images: List[Image] = None) to simplify calls when some modalities are unused.

Suggested change

prompt: str,

image: Image,

images: List[Image],

audio: Audio,

audios: List[Audio],

video: Video,

videos: List[Video],

messages: List[Dict],

prompt: str = '',

image: Image = None,

images: List[Image] = [],

audio: Audio = None,

audios: List[Audio] = [],

video: Video = None,

videos: List[Video] = [],

messages: List[Dict] = [],

Copilot · 2025-05-22T14:16:37Z

 import json
 import math
 import operator


[nitpick] Remove unused imports (json, math, operator) to reduce clutter, unless they are used elsewhere in this module.

Suggested change

import json

import math

import operator

Copilot · 2025-05-22T14:16:38Z

            raise ValueError("Image has no bytes")
        return PILImage.open(io.BytesIO(self.proto.base64))

+    def to_base64_str(self) -> str:


[nitpick] The to_base64_str implementation is duplicated across Image, Audio, and Video; consider extracting a common mixin or utility to avoid code duplication.

luv-bansal · 2025-05-23T06:33:17Z

+    def to_base64_str(self) -> str:
+        if isinstance(self.proto.base64, bytes):
+            return self.proto.base64.decode('utf-8')
+        elif isinstance(self.proto.base64, str):
+            return self.proto.base64
+        elif not self.proto.base64:
+            raise ValueError("Audio has no bytes")
+


similarly here for all to_base64_str methods

luv-bansal

I feel like to_base64_str implementation is currently wrong, because I'm sure self.proto.base64 is bytes and not a base64-encoded bytes. I know this confusion in naming is for long time in Clarifai

github-actions · 2025-05-23T11:47:31Z

Package	Line Rate	Health
clarifai	43%	❌
clarifai.cli	43%	❌
clarifai.client	71%	➖
clarifai.client.auth	74%	➖
clarifai.constants	100%	✔
clarifai.datasets	100%	✔
clarifai.datasets.export	80%	✔
clarifai.datasets.upload	75%	➖
clarifai.datasets.upload.loaders	37%	❌
clarifai.models	100%	✔
clarifai.modules	0%	❌
clarifai.rag	72%	➖
clarifai.runners	10%	❌
clarifai.runners.models	57%	➖
clarifai.runners.utils	56%	➖
clarifai.runners.utils.data_types	72%	➖
clarifai.schema	100%	✔
clarifai.urls	75%	➖
clarifai.utils	73%	➖
clarifai.utils.evaluation	67%	➖
clarifai.workflows	94%	✔
Summary	65% (6086 / 9427)	➖

Minimum allowed line rate is 50%

simplified client wrapper functions

f70e929

mogith-pn requested review from Copilot and luv-bansal and removed request for Copilot April 23, 2025 15:58

Copilot AI reviewed Apr 23, 2025

View reviewed changes

fixed imports

9e27cf1

mogith-pn requested review from sanjaychelliah and srikanthbachala20 May 2, 2025 09:19

sanjaychelliah approved these changes May 15, 2025

View reviewed changes

luv-bansal reviewed May 15, 2025

View reviewed changes

luv-bansal reviewed May 22, 2025

View reviewed changes

mogith-pn and others added 5 commits May 22, 2025 17:36

moved functions to openai_convertor utils

a50df12

resolved conflicts for openai convertors

b834763

Merge branch 'master' into PR-500-simplify-client-wrapper

c35a161

added downloads for audio.url and video.url

50268a3

fixed imports for openai convertor

6068cec

mogith-pn requested review from Copilot and luv-bansal May 22, 2025 14:13

Copilot AI reviewed May 22, 2025

View reviewed changes

luv-bansal reviewed May 23, 2025

View reviewed changes

Comment thread clarifai/runners/utils/data_types/data_types.py

luv-bansal reviewed May 23, 2025

View reviewed changes

Comment thread clarifai/runners/utils/openai_convertor.py Outdated

added failback for bytes file

2c36206

mogith-pn requested a review from luv-bansal May 23, 2025 09:02

luv-bansal reviewed May 23, 2025

View reviewed changes

Comment thread clarifai/runners/utils/data_types/data_types.py

luv-bansal approved these changes May 23, 2025

View reviewed changes

mogith-pn added 2 commits May 23, 2025 16:38

Added backoff conditions

230eea1

added changelog for 11.4.3

6ece438

Merge branch 'master' into PR-500-simplify-client-wrapper

98ec03d

mogith-pn requested a review from luv-bansal May 23, 2025 11:35

mogith-pn and others added 2 commits May 23, 2025 17:08

fix lint in __init__.py

d84da0e

fix lint for 11.4.3 version

fba689e

luv-bansal approved these changes May 23, 2025

View reviewed changes

mogith-pn merged commit 681d6a7 into master May 23, 2025
9 checks passed

mogith-pn deleted the PR-500-simplify-client-wrapper branch May 23, 2025 12:09

		return True


		def build_openai_chat_format(prompt: str, image: Image, images: List[Image], audio: Audio,

Conversation

mogith-pn commented Apr 23, 2025

Why

How

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

luv-bansal left a comment

Choose a reason for hiding this comment

Uh oh!

luv-bansal May 15, 2025

Choose a reason for hiding this comment

Uh oh!

luv-bansal May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI May 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

luv-bansal May 23, 2025

Choose a reason for hiding this comment

Uh oh!

luv-bansal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

luv-bansal May 22, 2025 •

edited

Loading