Merge pull request #4 from silviabolanosp/sync

Sync

.github/workflows/promptflow-evals-e2e-test.yml

@@ -52,6 +52,8 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - uses: snok/install-poetry@v1
+        with:
+          version: 1.5.1
       - uses: actions/download-artifact@v4
         with:
           name: promptflow-evals

.github/workflows/promptflow-evals-unit-test.yml

@@ -45,6 +45,8 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - uses: snok/install-poetry@v1
+        with:
+          version: 1.5.1
       - uses: actions/download-artifact@v4
         with:
           name: promptflow-evals

docs/concepts/concept-connections.md

@@ -15,7 +15,7 @@ Prompt flow provides a variety of pre-built connections, including Azure Open AI
 | [Open AI](https://openai.com/)                               | LLM or Python                   |
 | [Cognitive Search](https://azure.microsoft.com/en-us/products/search) | Vector DB Lookup or Python      |
 | [Serp](https://serpapi.com/)                                 | Serp API or Python              |
-| [Serverless](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/deployments-overview#deploy-models-with-model-as-a-service)                                               | LLM or Python                   |
+| [Serverless](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/deployments-overview#deploy-models-with-model-as-a-service-maas)                                               | LLM or Python                   |
 | Custom                                                       | Python                          |
 
 By leveraging connections in prompt flow, you can easily establish and manage connections to external APIs and data sources, facilitating efficient data exchange and interaction within their AI applications.

docs/integrations/tools/azure-ai-language-tool.md

@@ -18,7 +18,7 @@ Azure AI Language enables users with task-oriented and optimized pre-trained or
 ## Requirements
 PyPI package: [`promptflow-azure-ai-language`](https://pypi.org/project/promptflow-azure-ai-language/).
 - For AzureML users: 
-    follow this [wiki](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-custom-tool-package-creation-and-usage?view=azureml-api-2#prepare-runtime), starting from `Prepare runtime`.
+    follow this [wiki](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-custom-tool-package-creation-and-usage?view=azureml-api-2#prepare-compute-session), starting from `Prepare compute session`.
 - For local users: 
     ```
     pip install promptflow-azure-ai-language

docs/integrations/tools/llmlingua-prompt-compression-tool.md

@@ -6,18 +6,18 @@ LLMLingua Prompt Compression tool enables you to speed up large language model's
 ## Requirements
 PyPI package: [`llmlingua-promptflow`](https://pypi.org/project/llmlingua-promptflow/).
 - For Azure users: 
-    follow [the wiki for AzureML](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-custom-tool-package-creation-and-usage?view=azureml-api-2#prepare-runtime) or [the wiki for AI Studio](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/prompt-flow-tools/prompt-flow-tools-overview#custom-tools), starting from `Prepare runtime`.
+    follow [the wiki for AzureML](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-custom-tool-package-creation-and-usage?view=azureml-api-2#prepare-compute-session) or [the wiki for AI Studio](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/prompt-flow-tools/prompt-flow-tools-overview#custom-tools) to prepare the compute session.
 - For local users: 
     ```
     pip install llmlingua-promptflow
     ```
     You may also want to install the [Prompt flow for VS Code extension](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow).
 
 ## Prerequisite
-Create a MaaS deployment for large language model in Azure model catalog. Take the Llama model as an example, you can learn how to deploy and consume Meta Llama models with model as a service by  [the guidance for Azure AI Studio](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-llama?tabs=llama-three#deploy-meta-llama-models-with-pay-as-you-go) 
+Create a MaaS deployment for large language model in Azure model catalog. Take the Llama model as an example, you can learn how to deploy and consume Meta Llama models with model as a service by  [the guidance for Azure AI Studio](https://learn.microsoft.com/azure/ai-studio/how-to/deploy-models-llama?tabs=llama-three#deploy-meta-llama-models-as-a-serverless-api) 
 or
 [the guidance for Azure Machine Learning Studio
-](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-models-llama?view=azureml-api-2&tabs=llama-three#deploy-meta-llama-models-with-pay-as-you-go).
+](https://learn.microsoft.com/azure/machine-learning/how-to-deploy-models-llama?view=azureml-api-2&tabs=llama-three#deploy-meta-llama-models-with-pay-as-you-go).
 
 ## Inputs
 

docs/reference/tools-reference/llm-tool.md

@@ -25,7 +25,7 @@ Create OpenAI resources, Azure OpenAI resources or MaaS deployment with the LLM
 
 - **MaaS deployment**
 
-    Create MaaS deployment for models in Azure AI Studio model catalog with [instruction](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/deployments-overview#deploy-models-with-model-as-a-service)
+    Create MaaS deployment for models in Azure AI Studio model catalog with [instruction](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/deployments-overview#deploy-models-with-model-as-a-service-maas)
 
     You can create serverless connection to use this MaaS deployment.
 

src/promptflow-core/promptflow/core/_flow.py

@@ -16,6 +16,7 @@
 from promptflow.core._errors import MissingRequiredInputError
 from promptflow.core._model_configuration import PromptyModelConfiguration
 from promptflow.core._prompty_utils import (
+    _get_image_obj,
     convert_model_configuration_to_connection,
     convert_prompt_template,
     format_llm_response,
@@ -391,6 +392,11 @@ def _resolve_inputs(self, input_values):
             resolved_inputs[input_name] = input_values.get(input_name, value.get("default", None))
         if missing_inputs:
             raise MissingRequiredInputError(f"Missing required inputs: {missing_inputs}")
+
+        # Resolve image input
+        for k, v in resolved_inputs.items():
+            if isinstance(v, str):
+                resolved_inputs[k] = _get_image_obj(v, working_dir=self.path.parent)
         return resolved_inputs
 
     def _get_input_signature(self):
@@ -497,7 +503,9 @@ def estimate_token_count(self, *args, **kwargs):
             raise UserErrorException("Max_token needs to be integer.")
         elif response_max_token <= 1:
             raise UserErrorException(f"{response_max_token} is less than the minimum of max_tokens.")
-        total_token = num_tokens_from_messages(prompt, self._model._model) + (response_max_token or 0)
+        total_token = num_tokens_from_messages(prompt, self._model._model, working_dir=self.path.parent) + (
+            response_max_token or 0
+        )
         return total_token
 
 

src/promptflow-core/promptflow/core/_prompty_utils.py

@@ -13,6 +13,7 @@
 from openai import APIConnectionError, APIStatusError, APITimeoutError, BadRequestError, OpenAIError, RateLimitError
 
 from promptflow._utils.logger_utils import LoggerFactory
+from promptflow._utils.multimedia_utils import MIME_PATTERN, ImageProcessor
 from promptflow._utils.yaml_utils import load_yaml
 from promptflow.contracts.types import PromptTemplate
 from promptflow.core._connection import AzureOpenAIConnection, OpenAIConnection, _Connection
@@ -137,7 +138,8 @@ def convert_prompt_template(template, inputs, api):
             template_content=prompt, trim_blocks=True, keep_trailing_newline=True, **inputs
         )
     else:
-        rendered_prompt = build_messages(prompt=prompt, **inputs)
+        reference_images = find_referenced_image_set(inputs)
+        rendered_prompt = build_messages(prompt=prompt, images=reference_images, **inputs)
     return rendered_prompt
 
 
@@ -284,7 +286,7 @@ def format_stream(llm_response):
     return result
 
 
-def num_tokens_from_messages(messages, model):
+def num_tokens_from_messages(messages, model, working_dir):
     """Return the number of tokens used by a list of messages."""
     # Ref: https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken#6-counting-tokens-for-chat-completions-api-calls  # noqa: E501
     try:
@@ -307,10 +309,10 @@ def num_tokens_from_messages(messages, model):
         tokens_per_name = -1  # if there's a name, the role is omitted
     elif "gpt-3.5-turbo" in model or "gpt-35-turbo":
         logger.warning("gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
-        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613")
+        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613", working_dir=working_dir)
     elif "gpt-4" in model:
         logger.warning("gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
-        return num_tokens_from_messages(messages, model="gpt-4-0613")
+        return num_tokens_from_messages(messages, model="gpt-4-0613", working_dir=working_dir)
     else:
         raise NotImplementedError(
             f"num_tokens_from_messages() is not implemented for model {model}. "
@@ -321,13 +323,81 @@ def num_tokens_from_messages(messages, model):
     for message in messages:
         num_tokens += tokens_per_message
         for key, value in message.items():
-            num_tokens += len(encoding.encode(value))
+            if isinstance(value, str):
+                num_tokens += len(encoding.encode(value))
+            elif isinstance(value, list):
+                for item in value:
+                    value_type = item.get("type", "text")
+                    if value_type == "text":
+                        # Calculate content tokens
+                        num_tokens += len(encoding.encode(item["text"]))
+                    elif value_type == "image_url":
+                        image_content = item["image_url"]["url"]
+                        if ImageProcessor.is_url(image_content):
+                            image_obj = ImageProcessor.create_image_from_url(image_content)
+                            num_tokens += _num_tokens_for_image(image_obj.to_base64())
+                        elif ImageProcessor.is_base64(image_content):
+                            image_obj = ImageProcessor.create_image_from_base64(image_content)
+                            num_tokens += _num_tokens_for_image(image_obj.to_base64())
+                        else:
+                            # Calculate image input as content
+                            num_tokens += len(encoding.encode(item["image_url"]["url"]))
             if key == "name":
                 num_tokens += tokens_per_name
     num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
     return num_tokens
 
 
+def _get_image_obj(image_str, working_dir):
+    mime_pattern_with_content = MIME_PATTERN.pattern[:-1] + r":\s*(.*)$"
+    match = re.match(mime_pattern_with_content, image_str)
+    if match:
+        mine_type, image_type, image_content = f"image/{match.group(1)}", match.group(2), match.group(3)
+        if image_type == "path":
+            if not Path(image_content).is_absolute():
+                image_content = Path(working_dir) / image_content
+            if not Path(image_content).exists():
+                logger.warning(f"Cannot find the image path {image_content}, it will be regarded as {type(image_str)}.")
+            return ImageProcessor.create_image_from_file(image_content, mine_type)
+        elif image_type == "base64":
+            return ImageProcessor.create_image_from_base64(image_content, mine_type)
+        elif image_type == "url":
+            return ImageProcessor.create_image_from_url(image_content, mine_type)
+        else:
+            logger.warning(f"Invalid mine type {mine_type}, it will be regarded as {type(image_str)}.")
+    return image_str
+
+
+def _num_tokens_for_image(base64_str: str):
+    """calculate token of image input"""
+    # https://platform.openai.com/docs/guides/vision/calculating-costs
+    import base64
+    from io import BytesIO
+    from math import ceil
+
+    from PIL import Image
+
+    imgdata = base64.b64decode(base64_str)
+    image = Image.open(BytesIO(imgdata))
+    width, height = image.size
+    if width > 2048 or height > 2048:
+        aspect_ratio = width / height
+        if aspect_ratio > 1:
+            width, height = 2048, int(2048 / aspect_ratio)
+        else:
+            width, height = int(2048 * aspect_ratio), 2048
+
+    if width >= height and height > 768:
+        width, height = int((768 / height) * width), 768
+    elif height > width and width > 768:
+        width, height = 768, int((768 / width) * height)
+
+    tiles_width = ceil(width / 512)
+    tiles_height = ceil(height / 512)
+    image_tokens = 85 + 170 * (tiles_width * tiles_height)
+    return image_tokens
+
+
 def resolve_references(origin, base_path=None):
     """Resolve all reference in the object."""
     if isinstance(origin, str):
@@ -415,6 +485,15 @@ def merge_escape_mapping_of_flow_inputs(self, _inputs_to_escape: list, **kwargs)
         self.escaped_mapping.update(flow_inputs_escape_dict)
 
 
+def convert_to_chat_list(obj):
+    if isinstance(obj, dict):
+        return {key: convert_to_chat_list(value) for key, value in obj.items()}
+    elif isinstance(obj, list):
+        return ChatInputList([convert_to_chat_list(item) for item in obj])
+    else:
+        return obj
+
+
 def normalize_connection_config(connection):
     """
     Normalizes the configuration of a given connection object for compatibility.

src/promptflow-core/promptflow/core/_serving/app_base.py

@@ -117,6 +117,7 @@ def init_invoker_if_not_exist(self):
             storage=DummyRunStorage(),
             credential=self.credential,
             init_kwargs=self.init,
+            logger=self.logger,
         )
         # why we need to update bonded executable flow?
         self.flow = self.flow_invoker.flow

src/promptflow-core/promptflow/core/_serving/extension/default_extension.py

@@ -90,6 +90,7 @@ def get_flow_monitor(self, ctx_data_provider: ContextDataProvider) -> FlowMonito
         custom_dimensions = self.get_metrics_common_dimensions()
         metric_exporters = OTelExporterProviderFactory.get_metrics_exporters(self.logger, self.extension_type)
         trace_exporters = OTelExporterProviderFactory.get_trace_exporters(self.logger, self.extension_type)
+        log_exporters = OTelExporterProviderFactory.get_log_exporters(self.logger, self.extension_type)
         self.flow_monitor = FlowMonitor(
             self.logger,
             self.get_flow_name(),
@@ -98,6 +99,7 @@ def get_flow_monitor(self, ctx_data_provider: ContextDataProvider) -> FlowMonito
             custom_dimensions,
             metric_exporters,
             trace_exporters,
+            log_exporters,
         )  # noqa: E501
         return self.flow_monitor