Prompt Studio - Summarize, Output Analyzer and other UI Improvements by Deepak-Kesavan · Pull Request #9 · Zipstack/unstract

Deepak-Kesavan · 2024-02-27T12:59:12Z

What

Persist the metadata to handle summarization.
Persist the metadata to handle evaluation.
Component to display raw and summarized text.
Output analyzer.
UI improvements and bug fixes.

...

Why

NA
...

How

NA
...

Relevant Docs

NA

Related Issues or PRs

NA

Dependencies Versions / Env Variables

NA

Notes on Testing

NA
...

Screenshots

PDF view along with the prompt output for the raw content:

Summarize view along with the prompt output for the summarized content:

Settings modal for summarize:

Combined output with handled JSON formatting:

Output Analyzer:

...

Checklist

I have read and understood the Contribution Guidelines.

…t analyzer in the UI

…-summarize

…/unstract into feature/prompt-studio-summarize

hari-kuriakose

@tahierhussain Migrations look good overall.

muhammad-ali-e

LGTM

chandrasekharan-zipstack

BE changes LGTM, left a comment around plugin loading which could be addressed

harini-venkataraman · 2024-02-28T15:36:38Z

Apart from Minor comments, changes LGTM.

mahmoodmohaghegh · 2025-02-15T01:21:23Z

4872fd4

Prompt Studio - Summarize, Output Analyzer and other UI Improvements

… wrappers Address review comments on PR #1886: - #9 (typing): call_with_retry / acall_with_retry / iter_with_retry previously returned `object`, erasing caller type info. Add PEP 695 generics so the return type flows from the wrapped callable: acall_with_retry now takes Callable[[], Awaitable[T]] and iter_with_retry takes Callable[[], Iterable[T]] -> Generator[T, ...]. - #11 / #13 (DRY): `_pop_retry_params` in embedding.py and `_disable_litellm_retry` in llm.py were identical logic. Lift to shared `pop_litellm_retry_kwargs` helper in retry_utils.py and delete both methods. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* [FIX] Unified retry for LLM and embedding providers litellm's retry only works for SDK-based providers (OpenAI/Azure). httpx-based providers (Anthropic, Vertex, Bedrock, Mistral) and ALL embedding calls silently ignore max_retries. This adds self-managed retry with exponential backoff at the SDK layer, disabling litellm's own retry entirely for consistency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [REFACTOR] DRY retry logic into reusable call_with_retry utilities Move retry loops out of LLM/Embedding classes into generic call_with_retry, acall_with_retry, and iter_with_retry functions in retry_utils.py. Both classes now call these directly instead of maintaining their own retry helper methods. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [FIX] Consolidate retry logic, expose max_retries for all adapters - Extract _get_retry_delay() shared helper to eliminate duplicated retry decision logic across call_with_retry, acall_with_retry, iter_with_retry, and retry_with_exponential_backoff - Add num_retries=0 to embedding._pop_retry_params() to fully disable litellm's internal retry for embedding calls - Expose max_retries in UI JSON schemas for embedding adapters (OpenAI, Azure, VertexAI, Ollama) and Ollama LLM — previously the field existed in Pydantic models but wasn't shown to users, silently defaulting to 0 retries - Add debug logging to LLM and Embedding retry parameter extraction - Clarify docstrings distinguishing is_retryable_litellm_error() from is_retryable_error() (different exception hierarchies) - Remove stale noqa: C901 from simplified retry_with_exponential_backoff Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [FIX] Set max_retries default to 3 for all embedding and Ollama LLM adapters Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [FIX] Address greptile review: fix shadowed ConnectionError, use MRO check - Fix `requests.ConnectionError` shadowing Python's builtin `ConnectionError` in `is_retryable_litellm_error()` — rename import to `RequestsConnectionError` and use `builtins.ConnectionError` / `builtins.TimeoutError` explicitly - Use `__mro__`-based class name check instead of `type(error).__name__` to also catch subclasses of retryable error types - P1 (num_retries not zeroed) was already fixed in prior commit Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * [FIX] Address CodeRabbit review: add APITimeoutError, validate max_retries - Add APITimeoutError to _RETRYABLE_ERROR_NAMES for explicit OpenAI SDK timeout coverage - Add _validate_max_retries() guard to call_with_retry, acall_with_retry, iter_with_retry to fail fast on negative values instead of silently returning None Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * UN-3344 [FIX] Reduce cognitive complexity and remove useless except clause Address SonarCloud findings on PR #1886: - S3776: Flatten retry_with_exponential_backoff.wrapper by moving the success logging + return out of the try block and using `continue` in the retry path, so the except branch only handles the give-up case. - S2737: Drop the `except Exception: raise` clause — it was a no-op that added complexity without changing behavior (non-matching exceptions propagate naturally). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * UN-3344 [FIX] Extract retry loop to top-level helper to drop cognitive complexity Sonar still flagged retry_with_exponential_backoff at complexity 16 after the previous flatten. Nested def decorator / def wrapper counted against the outer function's score. Move the retry body to a module-level _invoke_with_retries helper so the decorator factory just delegates, bringing the outer function well under the 15 threshold. Behavior is unchanged — all paths (success, retry, give-up, non-retryable propagate) are preserved and covered by the existing SDK1 tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * UN-3344 [FIX] Honor Retry-After, close stream gen on retry, share give-up log Address review comments on PR #1886: - #10 (resource leak): close the generator returned by fn() before retrying in iter_with_retry — otherwise streaming providers leak an in-flight HTTP socket until GC. - #12 (behavioral regression): when we zero out SDK/wrapper retries we also lose the OpenAI SDK's native Retry-After handling on 429/503. _get_retry_delay now checks error.response.headers["retry-after"] and uses that value ahead of exponential backoff. HTTP-date form is not parsed; those fall back to backoff. - #8 (observability gap): move the "Giving up ... after N attempt(s)" log into _get_retry_delay so all four retry helpers (call_with_retry, acall_with_retry, iter_with_retry, decorator) share the same exhaustion signal. Previously only the decorator path logged it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * UN-3344 [REFACTOR] Share retry-kwargs helper and add TypeVar to retry wrappers Address review comments on PR #1886: - #9 (typing): call_with_retry / acall_with_retry / iter_with_retry previously returned `object`, erasing caller type info. Add PEP 695 generics so the return type flows from the wrapped callable: acall_with_retry now takes Callable[[], Awaitable[T]] and iter_with_retry takes Callable[[], Iterable[T]] -> Generator[T, ...]. - #11 / #13 (DRY): `_pop_retry_params` in embedding.py and `_disable_litellm_retry` in llm.py were identical logic. Lift to shared `pop_litellm_retry_kwargs` helper in retry_utils.py and delete both methods. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: ali <117142933+muhammad-ali-e@users.noreply.github.com>

tahierhussain and others added 12 commits February 27, 2024 03:56

Added /extract and /summarize path

4872fd4

Handled the logic to toggle between extract and summarize

fe3b205

Persist the evaluation data inside the output manager

498b42a

Handled summarize view, raw view, persistence of eval data, and outpu…

f71c791

…t analyzer in the UI

Merge remote-tracking branch 'origin/main' into feature/prompt-studio…

ec5c932

…-summarize

UI improvements and bug fixes

0f01042

Handled the state updation for the form values in SelectLlmProfileModal

f26639c

Merge branch 'feature/prompt-studio-summarize' of github.com:Zipstack…

b76c75f

…/unstract into feature/prompt-studio-summarize

Fixed the popup issue to display the error message

d60d5cf

Merge branch 'feature/prompt-studio-summarize' of github.com:Zipstack…

c7c07ed

…/unstract into feature/prompt-studio-summarize

Added migration files for summarize and eval

ddfce88

Added support for processor plugins

daef5ce

Deepak-Kesavan assigned tahierhussain and Deepak-Kesavan Feb 27, 2024

Deepak-Kesavan changed the title ~~Prompt studio summarize~~ Prompt Studio - Summarize, Output Analyzer and other UI Improvements Feb 27, 2024

Fixed scroll issue in the output analyzer

a48175f

tahierhussain requested review from chandrasekharan-zipstack, gaya3-zipstack, hari-kuriakose, harini-venkataraman, jaseemjaskp, mohamed-siddhiq, muhammad-ali-e, nehabagdia and vishnuszipstack February 27, 2024 14:31

nehabagdia approved these changes Feb 27, 2024

View reviewed changes

hari-kuriakose approved these changes Feb 27, 2024

View reviewed changes

muhammad-ali-e approved these changes Feb 27, 2024

View reviewed changes

chandrasekharan-zipstack approved these changes Feb 28, 2024

View reviewed changes

Comment thread backend/file_management/file_management_helper.py Outdated

Comment thread backend/prompt_studio/prompt_studio_core/models.py

Comment thread backend/prompt_studio/prompt_studio_core/prompt_studio_helper.py Outdated

Comment thread backend/prompt_studio/prompt_studio_core/views.py

Merge branch 'main' into feature/prompt-studio-summarize

a1ef571

Deepak-Kesavan added 2 commits February 28, 2024 14:31

Updated sdk version and resolved review comments

a41f5f8

Updated migrations

5c1e02d

harini-venkataraman reviewed Feb 28, 2024

View reviewed changes

Comment thread backend/file_management/file_management_helper.py

Deepak-Kesavan added 2 commits February 28, 2024 20:23

Added processor directory

6744598

Updated sdk version to 0.12.1

9309da5

Deepak-Kesavan marked this pull request as ready for review February 28, 2024 15:16

harini-venkataraman reviewed Feb 28, 2024

View reviewed changes

Comment thread backend/prompt_studio/prompt_studio_core/models.py

harini-venkataraman approved these changes Feb 28, 2024

View reviewed changes

nehabagdia merged commit 1798a19 into main Feb 28, 2024

nehabagdia deleted the feature/prompt-studio-summarize branch February 28, 2024 16:28

pk-zipstack pushed a commit that referenced this pull request Aug 20, 2025

Merge pull request #9 from Zipstack/feature/prompt-studio-summarize

588cf8d

Prompt Studio - Summarize, Output Analyzer and other UI Improvements

muhammad-ali-e mentioned this pull request Apr 28, 2026

UN-3435 [FIX] Bind request_id and trace context to worker logs #1932

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prompt Studio - Summarize, Output Analyzer and other UI Improvements#9

Prompt Studio - Summarize, Output Analyzer and other UI Improvements#9
nehabagdia merged 18 commits intomainfrom
feature/prompt-studio-summarize

Deepak-Kesavan commented Feb 27, 2024 •

edited by tahierhussain

Loading

Uh oh!

hari-kuriakose left a comment

Uh oh!

muhammad-ali-e left a comment

Uh oh!

chandrasekharan-zipstack left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harini-venkataraman commented Feb 28, 2024

Uh oh!

mahmoodmohaghegh commented Feb 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

Deepak-Kesavan commented Feb 27, 2024 • edited by tahierhussain Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Relevant Docs

NA

Related Issues or PRs

NA

Dependencies Versions / Env Variables

NA

Notes on Testing

Screenshots

Checklist

Uh oh!

hari-kuriakose left a comment

Choose a reason for hiding this comment

Uh oh!

muhammad-ali-e left a comment

Choose a reason for hiding this comment

Uh oh!

chandrasekharan-zipstack left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harini-venkataraman commented Feb 28, 2024

Uh oh!

mahmoodmohaghegh commented Feb 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Deepak-Kesavan commented Feb 27, 2024 •

edited by tahierhussain

Loading