Skip to content

Test langsmith flake fix (1000x per matrix) #1510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
xumaple wants to merge 2 commits into
base: main
Choose a base branch
from
+21 −15
Draft

Test langsmith flake fix (1000x per matrix) #1510

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e754847
DO NOT MERGE: stress-test langsmith flake fix (1000x per matrix)
xumaple May 8, 2026
ea8bc4e
DO NOT MERGE: negative control — restore max_cached_workflows=0
xumaple May 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 18 additions & 15 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:
jobs:
# Build and test the project
build-lint-test:
timeout-minutes: 30
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -58,22 +58,25 @@ jobs:
- uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8
- run: uv tool install poethepoet
- run: uv sync --all-extras
- run: poe bridge-lint
if: ${{ matrix.clippyLinter }}
# [STRESS TEST BRANCH] bridge-lint disabled
# - run: poe bridge-lint
# if: ${{ matrix.clippyLinter }}
- run: poe build-develop
- run: poe lint
# [STRESS TEST BRANCH] poe lint disabled
# - run: poe lint
- run: mkdir junit-xml
- run: poe test ${{matrix.pytestExtraArgs}} -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}.xml
timeout-minutes: 15
# Time skipping doesn't yet support ARM
- if: ${{ !endsWith(matrix.os, '-arm') }}
run: poe test ${{matrix.pytestExtraArgs}} -s --workflow-environment time-skipping --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--time-skipping.xml
timeout-minutes: 10
- if: ${{ matrix.openaiTestTarget && (github.event.pull_request.head.repo.full_name == '' || github.event.pull_request.head.repo.full_name == 'temporalio/sdk-python') }}
run: poe test tests/contrib/openai_agents/test_openai.py ${{matrix.pytestExtraArgs}} -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--openai.xml
timeout-minutes: 10
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
# [STRESS TEST BRANCH] Run only the flaky test 1000x with no rerun hiding
- run: uv run pytest tests/contrib/langsmith/test_integration.py::TestBuiltinQueryFiltering::test_temporal_prefixed_query_not_traced -n auto --dist=worksteal -p no:rerunfailures -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}.xml
timeout-minutes: 30
# [STRESS TEST BRANCH] Time-skipping and openai steps disabled for this branch
# - if: ${{ !endsWith(matrix.os, '-arm') }}
# run: poe test ${{matrix.pytestExtraArgs}} -s --workflow-environment time-skipping --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--time-skipping.xml
# timeout-minutes: 10
# - if: ${{ matrix.openaiTestTarget && (github.event.pull_request.head.repo.full_name == '' || github.event.pull_request.head.repo.full_name == 'temporalio/sdk-python') }}
# run: poe test tests/contrib/openai_agents/test_openai.py ${{matrix.pytestExtraArgs}} -s --junit-xml=junit-xml/${{ matrix.python }}--${{ matrix.os }}--openai.xml
# timeout-minutes: 10
# env:
# OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- name: "Upload junit-xml artifacts"
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
if: always()
Expand Down
3 changes: 3 additions & 0 deletions tests/contrib/langsmith/test_integration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1224,9 +1224,11 @@ def my_query(self) -> str:
class TestBuiltinQueryFiltering:
"""Verifies __temporal_ prefixed queries are not traced."""

@pytest.mark.parametrize("_iteration", range(1000))
async def test_temporal_prefixed_query_not_traced(
self,
client: Client,
_iteration: int,
) -> None:
"""__temporal_workflow_metadata query should not produce a trace,
but user-defined queries should still be traced.
Expand All @@ -1235,6 +1237,7 @@ async def test_temporal_prefixed_query_not_traced(
client-side QueryWorkflow traces, isolating the test to
worker-side HandleQuery traces only.
"""
del _iteration

task_queue = f"query-filter-{uuid.uuid4()}"
collector = InMemoryRunCollector()
Expand Down
Loading