Open
Description
Describe the bug
Flow considers only the flow file to create the hash used for caching: https://github.com/microsoft/promptflow/blob/promptflow_1.16.1/src/promptflow-devkit/promptflow/_sdk/entities/_flows/base.py#L166-L169.
If we have two flows with the exactly same flow.dag.yaml, but different tools, it may run the wrong flow.
How To Reproduce the bug
(see the test reproducing the issue in Additional context section below)
- Create a flow with at least one tool.
- Create a second flow with the exactly same
flow.dag.yamlbut different tools. - Invoke both flows using Python API.
Current behavior: Only the first loaded flow runs twice.
Expected behavior: Two different flows are loaded and run.
Running Information:
- Promptflow Package Version:
{ "promptflow": "1.16.1", "promptflow-core": "1.16.1", "promptflow-devkit": "1.16.1", "promptflow-tracing": "1.16.1" } Executable '/home/user/workspace/test-promptflow/.venv/bin/python' Python (Linux) 3.12.1 (main, Dec 12 2023, 10:34:48) [GCC 10.2.1 20210110] - Operating System:
Debian GNU/Linux 11 (bullseye) - Python Version:
3.12.1
Additional context
Test reproducing the issue:
import tempfile
from pathlib import Path
from textwrap import dedent
import pytest
from promptflow.client import load_flow
@pytest.fixture
def flow_yaml_content():
return dedent(
"""
$schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
inputs:
text:
type: string
outputs:
text:
type: string
reference: ${transform}
nodes:
- name: transform
type: python
source:
type: code
path: transform.py
inputs:
text: ${inputs.text}"""
)
@pytest.fixture
def lower_flow_dir(flow_yaml_content):
with tempfile.TemporaryDirectory() as lower_flow_dir:
lower_flow_dir = Path(lower_flow_dir)
with open(lower_flow_dir / 'flow.dag.yaml', 'w') as f:
f.write(flow_yaml_content)
with open(lower_flow_dir / "transform.py", "w") as f:
f.write(
dedent(
"""
from promptflow import tool
@tool
def lower(text):
return text.lower()"""
)
)
yield lower_flow_dir
@pytest.fixture
def upper_flow_dir(flow_yaml_content):
with tempfile.TemporaryDirectory() as upper_flow_dir:
upper_flow_dir = Path(upper_flow_dir)
with open(upper_flow_dir / 'flow.dag.yaml', 'w') as f:
f.write(flow_yaml_content)
with open(upper_flow_dir / "transform.py", "w") as f:
f.write(
dedent(
"""
from promptflow import tool
@tool
def upper(text):
return text.upper()"""
)
)
yield upper_flow_dir
def test_ignore_cache_when_tools_are_different(lower_flow_dir, upper_flow_dir):
assert load_flow(lower_flow_dir).invoke({"text": "TEST"}).output == {"text": "test"}
assert load_flow(upper_flow_dir).invoke({"text": "test"}).output == {"text": "TEST"}___________________________ test_ignore_cache_when_tools_are_different ___________________________
lower_flow_dir = PosixPath('/tmp/tmpiizohoz4'), upper_flow_dir = PosixPath('/tmp/tmp1wl2g71f')
def test_ignore_cache_when_tools_are_different(lower_flow_dir, upper_flow_dir):
assert load_flow(lower_flow_dir).invoke({"text": "TEST"}).output == {"text": "test"}
> assert load_flow(upper_flow_dir).invoke({"text": "test"}).output == {"text": "TEST"}
E AssertionError: assert {'text': 'test'} == {'text': 'TEST'}
E
E Differing items:
E {'text': 'test'} != {'text': 'TEST'}
E Use -v to get more diff
test_promptflow.py:79: AssertionError