Migrate act() to conversation-based architecture with Speaker pattern and add caching v2 features. #236

programminx-askui · 2026-03-03T12:49:12Z

Missing switch_callback

programminx-askui · 2026-03-03T12:33:26Z

Nice Example

-Original file line number
+Diff line change
@@ Expand Up / @@ -169,5 +169,6 @@ reports/ @@
     /askui_chat.db-shm
     /askui_chat.db-wal
     .cache/
+    .askui_cache/*
     bom.json

-Original file line number
+Diff line change
@@ Expand Up @@
 . **[Reporting](docs/08_reporting.md)** - Obtain agent logs as execution reports and summaries as test reports
 . **[Observability](docs/09_observability_telemetry_tracing.md)** - Monitor and debug agents
 . **[Extracting Data](docs/10_extracting_data.md)** - Extracting structured data from screenshots and files
+. **[Callbacks](docs/11_callbacks.md)** - Inject custom logic into the control loop
     **Official documentation:** [docs.askui.com](https://docs.askui.com)
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -90,6 +90,9 @@ Understand what data is collected and how to opt out. @@
     ### 10 - Extracting Data
     **Topics**: Using `get()`, file support (PDF. Excel, Word, CSV), structured data extraction, response schemas
+    ### 11 - Callbacks
+    **Topics**: Inject custom logic at different positions of the control loop through callbacks
     Extract information from screens and files using the `get()` method with Pydantic models.
     ## Additional Resources
@@ Expand Down @@

-Original file line number
+Diff line change
@@ -0,0 +1,82 @@
+    # Callbacks
+    Callbacks provide hooks into the agent's conversation lifecycle, similar to PyTorch Lightning's callback system. Use them for logging, monitoring, custom metrics, or extending agent behavior.
+    ## Usage
+    Subclass `ConversationCallback` and override the hooks you need:
+    ```python
+    from askui import ComputerAgent, ConversationCallback
+    class MetricsCallback(ConversationCallback):
+        def on_step_start(self, conversation, step_index):
+            print(f"Step {step_index} starting...")
+        def on_step_end(self, conversation, step_index, result):
+            print(f"Step {step_index} finished: {result.status}")
+    with ComputerAgent(callbacks=[MetricsCallback()]) as agent:
+        agent.act("Open the settings menu")
+    ```
+    ## Available Hooks
+    | Hook | When Called | Parameters |
+    |------|-------------|------------|
+    | `on_conversation_start` | After setup, before control loop | `conversation` |
+    | `on_conversation_end` | After control loop, before cleanup | `conversation` |
+    | `on_control_loop_start` | Before the iteration loop begins | `conversation` |
+    | `on_control_loop_end` | After the iteration loop ends | `conversation` |
+    | `on_step_start` | Before each step execution | `conversation`, `step_index` |
+    | `on_step_end` | After each step execution | `conversation`, `step_index`, `result` |
+    | `on_tool_execution_start` | Before tools are executed | `conversation`, `tool_names` |
+    | `on_tool_execution_end` | After tools are executed | `conversation`, `tool_names` |
+    ### Parameters
+    - **`conversation`**: The `Conversation` instance with access to messages, settings, and state
+    - **`step_index`**: Zero-based index of the current step
+    - **`result`**: `SpeakerResult` containing `status`, `messages_to_add`, and `usage`
+    - **`tool_names`**: List of tool names being executed
+    ## Example: Timing Callback
+    ```python
+    import time
+    from askui import ComputerAgent, ConversationCallback
+    class TimingCallback(ConversationCallback):
+        def __init__(self):
+            self.start_time = None
+            self.step_times = []
+        def on_conversation_start(self, conversation):
+            self.start_time = time.time()
+        def on_step_start(self, conversation, step_index):
+            self._step_start = time.time()
+        def on_step_end(self, conversation, step_index, result):
+            elapsed = time.time() - self._step_start
+            self.step_times.append(elapsed)
+            print(f"Step {step_index}: {elapsed:.2f}s")
+        def on_conversation_end(self, conversation):
+            total = time.time() - self.start_time
+            print(f"Total: {total:.2f}s across {len(self.step_times)} steps")
+    with ComputerAgent(callbacks=[TimingCallback()]) as agent:
+        agent.act("Search for documents")
+    ```
+    ## Multiple Callbacks
+    Pass multiple callbacks to combine behaviors:
+    ```python
+    with ComputerAgent(callbacks=[TimingCallback(), MetricsCallback()]) as agent:
+        agent.act("Complete the form")
+    ```
+    Callbacks are called in the order they are provided.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate act() to conversation-based architecture with Speaker pattern and add caching v2 features. #236

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

programminx-askui Mar 3, 2026

Uh oh!

programminx-askui Mar 3, 2026

Uh oh!

Uh oh!

-Original file line number
+Diff line change
@@ Expand Up @@
     ## Caching Strategies
-    The caching mechanism supports four strategies, configured via the `caching_settings` parameter in the `act()` method:
+    The caching mechanism supports three strategies, configured via the `caching_settings` parameter in the `act()` method:
-    - **`"no"`** (default): No caching is used. The agent executes normally without recording or replaying actions.
-    - **`"write"`**: Records all agent actions to a cache file for future replay.
-    - **`"read"`**: Provides tools to the agent to list and execute previously cached trajectories.
-    - **`"both"`**: Combines read and write modes - the agent can use existing cached trajectories and will also record new ones.
+    - **`None`** (default): No caching is used. The agent executes normally without recording or replaying actions.
+    - **`"record"`**: Records all agent actions to a cache file for future replay.
+    - **`"execute"`**: Provides tools to the agent to list and execute previously cached trajectories.
+    - **`"auto"`**: Combines execute and record modes - the agent can use existing cached trajectories and will also record new ones.
     ## Configuration
@@ Expand All / @@ -23,20 +23,20 @@ Caching is configured using the `CachingSettings` class: @@
     from askui.models.shared.settings import CachingSettings, CachedExecutionToolSettings
     caching_settings = CachingSettings(
-        strategy="write",        # One of: "read", "write", "both", "no"
+        strategy="record",       # One of: "execute", "record", "auto", or None
         cache_dir=".cache",      # Directory to store cache files
-        filename="my_test.json", # Filename for the cache file (optional for write mode)
+        filename="my_test.json", # Filename for the cache file (optional for record mode)
         execute_cached_trajectory_tool_settings=CachedExecutionToolSettings(
-            delay_time_between_action=0.5  # Delay in seconds between each cached action
+            delay_time_between_actions=0.5  # Delay in seconds between each cached action
         )
     )
     ```
     ### Parameters
-    - **`strategy`**: The caching strategy to use (`"read"`, `"write"`, `"both"`, or `"no"`).
+    - **`strategy`**: The caching strategy to use (`"execute"`, `"record"`, `"auto"`, or `None`).
     - **`cache_dir`**: Directory where cache files are stored. Defaults to `".cache"`.
-    - **`filename`**: Name of the cache file to write to or read from. If not specified in write mode, a timestamped filename will be generated automatically (format: `cached_trajectory_YYYYMMDDHHMMSSffffff.json`).
+    - **`filename`**: Name of the cache file to write to or read from. If not specified in record mode, a timestamped filename will be generated automatically (format: `cached_trajectory_YYYYMMDDHHMMSSffffff.json`).
     - **`execute_cached_trajectory_tool_settings`**: Configuration for the trajectory execution tool (optional). See [Execution Settings](#execution-settings) below.
     ### Execution Settings
@@ Expand All @@
     from askui.models.shared.settings import CachedExecutionToolSettings
     execution_settings = CachedExecutionToolSettings(
-        delay_time_between_action=0.5  # Delay in seconds between each action (default: 0.5)
+        delay_time_between_actions=0.5  # Delay in seconds between each action (default: 0.5)
     )
     ```
     #### Parameters
-    - **`delay_time_between_action`**: The time to wait (in seconds) between executing consecutive cached actions. This delay helps ensure UI elements can materialize before the next action is executed. Defaults to `0.5` seconds.
+    - **`delay_time_between_actions`**: The time to wait (in seconds) between executing consecutive cached actions. This delay helps ensure UI elements can materialize before the next action is executed. Defaults to `0.5` seconds.
     You can adjust this value based on your application's responsiveness:
     - For faster applications or quick interactions, you might use a smaller delay (e.g., `0.1` or `0.2` seconds)
     - For slower applications or complex UI updates, you might need a longer delay (e.g., `1.0` or `2.0` seconds)
     ## Usage Examples
-    ### Writing a Cache (Recording)
+    ### Recording a Cache
     Record agent actions to a cache file for later replay:
@@ Expand All / @@ -73,7 +73,7 @@ with ComputerAgent() as agent: @@
         agent.act(
             goal="Fill out the login form with username 'admin' and password 'secret123'",
             caching_settings=CachingSettings(
-                strategy="write", # you could also use "both" here
+                strategy="record", # you could also use "auto" here
                 cache_dir=".cache",
                 filename="login_test.json"
             )
@@ Expand All / @@ -82,7 +82,7 @@ with ComputerAgent() as agent: @@
     After execution, a cache file will be created at `.cache/login_test.json` containing all the tool use actions performed by the agent.
-    ### Reading from Cache (Replaying)
+    ### Executing from Cache (Replaying)
     Provide the agent with access to previously recorded trajectories:
@@ Expand All / @@ -94,13 +94,13 @@ with ComputerAgent() as agent: @@
         agent.act(
             goal="Fill out the login form",
             caching_settings=CachingSettings(
-                strategy="read", # you could also use "both" here
+                strategy="execute", # you could also use "auto" here
                 cache_dir=".cache"
             )
         )
     ```
-    When using `strategy="read"`, the agent receives two additional tools:
+    When using `strategy="execute"`, the agent receives two additional tools:
 . **`retrieve_available_trajectories_tool`**: Lists all available cache files in the cache directory
 . **`execute_cached_executions_tool`**: Executes a specific cached trajectory
@@ Expand All @@
     ### Referencing Cache Files in Goal Prompts
-    When using `strategy="read"` or `strategy="both"`, **you need to inform the agent about which cache files are available and when to use them**. This is done by including cache file information directly in your goal prompt.
+    When using `strategy="execute"` or `strategy="auto"`, **you need to inform the agent about which cache files are available and when to use them**. This is done by including cache file information directly in your goal prompt.
     #### Explicit Cache File References
@@ Expand All / @@ -126,7 +126,7 @@ with ComputerAgent() as agent: @@
             If the cache file "open_website_in_chrome.json" is available, please use it
             for this execution. It will open a new window in Chrome and navigate to the website.""",
             caching_settings=CachingSettings(
-                strategy="read",
+                strategy="execute",
                 cache_dir=".cache"
             )
         )
@@ Expand All / @@ -149,7 +149,7 @@ with ComputerAgent() as agent: @@
             Check if a cache file named "{test_id}.json" exists. If it does, use it to
             replay the test actions, then verify the results.""",
             caching_settings=CachingSettings(
-                strategy="read",
+                strategy="execute",
                 cache_dir="test_cache"
             )
         )
@@ Expand All / @@ -171,7 +171,7 @@ with ComputerAgent() as agent: @@
             Choose the most recent one if multiple are available, as it likely contains
             the most up-to-date interaction sequence.""",
             caching_settings=CachingSettings(
-                strategy="read",
+                strategy="execute",
                 cache_dir=".cache"
             )
         )
@@ Expand All / @@ -195,7 +195,7 @@ with ComputerAgent() as agent: @@
             After each cached execution, verify the step completed successfully before proceeding.""",
             caching_settings=CachingSettings(
-                strategy="read",
+                strategy="execute",
                 cache_dir=".cache"
             )
         )
@@ Expand All / @@ -219,10 +219,10 @@ with ComputerAgent() as agent: @@
         agent.act(
             goal="Fill out the login form",
             caching_settings=CachingSettings(
-                strategy="read",
+                strategy="execute",
                 cache_dir=".cache",
                 execute_cached_trajectory_tool_settings=CachedExecutionToolSettings(
-                    delay_time_between_action=1.0  # Wait 1 second between each action
+                    delay_time_between_actions=1.0  # Wait 1 second between each action
                 )
             )
         )
@@ Expand All / @@ -233,7 +233,7 @@ This is particularly useful when: @@
     - UI elements take time to become interactive after appearing
     - You're testing on slower hardware or environments
-    ### Using Both Strategies
+    ### Using Auto Strategy
     Enable both reading and writing simultaneously:
@@ Expand All / @@ -245,7 +245,7 @@ with ComputerAgent() as agent: @@
         agent.act(
             goal="Complete the checkout process",
             caching_settings=CachingSettings(
-                strategy="both",
+                strategy="auto",
                 cache_dir=".cache",
                 filename="checkout_test.json"
             )
@@ Expand Down Expand Up @@
     ## Limitations
     - **UI State Sensitivity**: Cached trajectories assume the UI is in the same state as when they were recorded. If the UI has changed, the replay may fail or produce incorrect results.
-    - **No on_message Callback**: When using `strategy="write"` or `strategy="both"`, you cannot provide a custom `on_message` callback, as the caching system uses this callback to record actions.
+    - **No on_message Callback**: When using `strategy="record"` or `strategy="auto"`, you cannot provide a custom `on_message` callback, as the caching system uses this callback to record actions.
     - **Verification Required**: After executing a cached trajectory, the agent should verify that the results are correct, as UI changes may cause partial failures.
     ## Example: Complete Test Workflow
@@ Expand All / @@ -340,7 +340,7 @@ with ComputerAgent() as agent: @@
         agent.act(
             goal="Navigate to the login page and log in with username 'testuser' and password 'testpass123'",
             caching_settings=CachingSettings(
-                strategy="write",
+                strategy="record",
                 cache_dir="test_cache",
                 filename="user_login.json"
             )
@@ Expand All / @@ -356,10 +356,10 @@ with ComputerAgent() as agent: @@
             the login sequence. It contains the steps to navigate to the login page and
             authenticate with the test credentials.""",
             caching_settings=CachingSettings(
-                strategy="read",
+                strategy="execute",
                 cache_dir="test_cache",
                 execute_cached_trajectory_tool_settings=CachedExecutionToolSettings(
-                    delay_time_between_action=1.0
+                    delay_time_between_actions=1.0
                 )
             )
         )
@@ Expand Down @@

Migrate act() to conversation-based architecture with Speaker pattern and add caching v2 features. #236

Are you sure you want to change the base?

Uh oh!

Migrate act() to conversation-based architecture with Speaker pattern and add caching v2 features. #236

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

programminx-askui Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

programminx-askui Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!