Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .adl-ignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ skills/take_screenshot.go
skills/execute_script.go
skills/handle_authentication.go
skills/wait_for_condition.go
skills/write_to_csv.go
internal/playwright/playwright.go

# Go dependency files
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/cd.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .github/workflows/ci.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion .releaserc.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions .well-known/agent.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 15 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Your automation solutions should be maintainable, efficient, and production-read
## Skills


This agent provides 8 skills:
This agent provides 9 skills:


### navigate_to_url
Expand Down Expand Up @@ -134,6 +134,13 @@ This agent provides 8 skills:
- **Output Schema**: Defined in agent configuration


### write_to_csv
- **Description**: Write structured data to CSV files with support for custom headers and file paths
- **Tags**: export, csv, data, file
- **Input Schema**: Defined in agent configuration
- **Output Schema**: Defined in agent configuration




## Server Configuration
Expand Down Expand Up @@ -239,6 +246,11 @@ curl -X POST http://localhost:8080/skills/wait_for_condition \
-H "Content-Type: application/json" \
-d '{"input": "your_input_here"}'

# Execute write_to_csv skill
curl -X POST http://localhost:8080/skills/write_to_csv \
-H "Content-Type: application/json" \
-d '{"input": "your_input_here"}'


```

Expand Down Expand Up @@ -286,6 +298,8 @@ docker run -p 8080:8080 browser-agent

│ └── wait_for_condition.go # Wait for specific conditions before proceeding with automation

│ └── write_to_csv.go # Write structured data to CSV files with support for custom headers and file paths

├── .well-known/ # Agent configuration
│ └── agent.json # Agent metadata
├── go.mod # Go module definition
Expand Down
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ browser-agent is an A2A (Agent-to-Agent) server implementing the [A2A Protocol](

### ADL-Generated Structure

The codebase is generated using ADL CLI 0.21.4 and follows a strict generation pattern:
The codebase is generated using ADL CLI 0.21.5 and follows a strict generation pattern:
- **Generated Files**: Marked with `DO NOT EDIT` headers - manual changes will be overwritten
- **Configuration Source**: `agent.yaml` - defines agent capabilities, skills, and metadata
- **Server Implementation**: Built on the ADK (Agent Development Kit) framework from `github.com/inference-gateway/adk`
Expand Down Expand Up @@ -82,6 +82,7 @@ The following skills are currently defined:
- **execute_script**: Execute custom JavaScript code in the browser context
- **handle_authentication**: Handle various authentication scenarios including basic auth, OAuth, and custom login forms
- **wait_for_condition**: Wait for specific conditions before proceeding with automation
- **write_to_csv**: Write structured data to CSV files with support for custom headers and file paths

To modify skills:
1. Update `agent.yaml` with skill definitions
Expand Down Expand Up @@ -117,7 +118,7 @@ Activate with: `flox activate` (if Flox is installed)

- **Generated Files**: Never manually edit files with "DO NOT EDIT" headers
- **Configuration Changes**: Always modify `agent.yaml` and regenerate
- **ADL Version**: Ensure ADL CLI 0.21.4 or compatible version for regeneration
- **ADL Version**: Ensure ADL CLI 0.21.5 or compatible version for regeneration
- **Port Configuration**: Default 8080, configurable via `A2A_PORT` or `A2A_SERVER_PORT`

## Debugging Tips
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ docker run -p 8080:8080 browser-agent
| `execute_script` | Execute custom JavaScript code in the browser context |args, return_value, script |
| `handle_authentication` | Handle various authentication scenarios including basic auth, OAuth, and custom login forms |login_url, password, password_selector, submit_selector, type, username, username_selector |
| `wait_for_condition` | Wait for specific conditions before proceeding with automation |condition, custom_function, selector, state, timeout |
| `write_to_csv` | Write structured data to CSV files with support for custom headers and file paths |append, data, filename, headers, include_headers |

## Configuration

Expand All @@ -61,13 +62,13 @@ The following custom configuration variables are available:
| Category | Variable | Description | Default |
|----------|----------|-------------|---------|
| **Browser** | `BROWSER_ARGS` | Args configuration | `[--disable-blink-features=AutomationControlled --disable-features=VizDisplayCompositor --no-first-run --disable-default-apps --disable-extensions --disable-plugins --disable-sync --disable-translate --hide-scrollbars --mute-audio --no-zygote --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-renderer-backgrounding --disable-ipc-flooding-protection]` |
| **Browser** | `BROWSER_DATA_DIR` | Data_dir configuration | `/tmp/playwright/artifacts` |
| **Browser** | `BROWSER_HEADER_ACCEPT` | Header_accept configuration | `text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7` |
| **Browser** | `BROWSER_HEADER_ACCEPT_ENCODING` | Header_accept_encoding configuration | `gzip, deflate, br` |
| **Browser** | `BROWSER_HEADER_ACCEPT_LANGUAGE` | Header_accept_language configuration | `en-US,en;q=0.9` |
| **Browser** | `BROWSER_HEADER_CONNECTION` | Header_connection configuration | `keep-alive` |
| **Browser** | `BROWSER_HEADER_DNT` | Header_dnt configuration | `1` |
| **Browser** | `BROWSER_HEADER_UPGRADE_INSECURE_REQUESTS` | Header_upgrade_insecure_requests configuration | `1` |
| **Browser** | `BROWSER_SCREENSHOTS_DIR` | Screenshots_dir configuration | `/tmp/screenshots` |
| **Browser** | `BROWSER_USER_AGENT` | User_agent configuration | `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36` |
| **Browser** | `BROWSER_VIEWPORT_HEIGHT` | Viewport_height configuration | `1080` |
| **Browser** | `BROWSER_VIEWPORT_WIDTH` | Viewport_width configuration | `1920` |
Expand Down
2 changes: 1 addition & 1 deletion Taskfile.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

40 changes: 39 additions & 1 deletion agent.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ spec:
header_dnt: "1"
header_connection: "keep-alive"
header_upgrade_insecure_requests: "1"
screenshots_dir: "/tmp/screenshots"
data_dir: "/tmp/playwright/artifacts"
args:
- "--disable-blink-features=AutomationControlled"
- "--disable-features=VizDisplayCompositor"
Expand Down Expand Up @@ -310,6 +310,44 @@ spec:
inject:
- logger
- playwright
- id: write_to_csv
name: write_to_csv
description: Write structured data to CSV files with support for custom headers and file paths
tags:
- export
- csv
- data
- file
schema:
type: object
properties:
data:
type: array
items:
type: object
description: Array of objects to write to CSV, each object represents a row
filename:
type: string
description: Name of the CSV file (without path, will be saved to configured data directory)
headers:
type: array
items:
type: string
description: Custom column headers for the CSV file (optional, will use object keys if not provided)
append:
type: boolean
description: Whether to append to existing file or create new file
default: false
include_headers:
type: boolean
description: Whether to include headers in the CSV output
default: true
required:
- data
- filename
inject:
- logger
- playwright
agent:
provider: ""
model: ""
Expand Down
4 changes: 2 additions & 2 deletions config/config.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions example/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Inference Gateway
DEEPSEEK_API_KEY=
GOOGLE_API_KEY=

# Agent
A2A_AGENT_CLIENT_PROVIDER=deepseek
A2A_AGENT_CLIENT_MODEL=deepseek-chat
2 changes: 0 additions & 2 deletions example/.env.gateway.example

This file was deleted.

1 change: 0 additions & 1 deletion example/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion example/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This script demonstrates how to use the Playwright automation framework to perfo
Configure the environment variables as needed:

```bash
cp .env.gateway.example .env.gateway
cp .env.example .env
```

** Add at least two providers, in this example Google and DeepSeek.
Expand Down
2 changes: 2 additions & 0 deletions example/artifacts/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 10 additions & 23 deletions example/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,8 @@ services:
ports:
- "8080:8080"
volumes:
- ./screenshots:/tmp/screenshots
- ./artifacts:/tmp/playwright/artifacts
environment:
BROWSER_SCREENSHOTS_DIR: /tmp/screenshots
BROWSER_USER_AGENT: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
BROWSER_VIEWPORT_WIDTH: "1920"
BROWSER_VIEWPORT_HEIGHT: "1080"
Expand Down Expand Up @@ -45,8 +44,8 @@ services:
A2A_SERVER_IDLE_TIMEOUT: 120s
A2A_SERVER_DISABLE_HEALTHCHECK_LOG: true
A2A_AGENT_CARD_FILE_PATH: .well-known/agent.json
A2A_AGENT_CLIENT_PROVIDER: google
A2A_AGENT_CLIENT_MODEL: models/gemini-2.5-flash
A2A_AGENT_CLIENT_PROVIDER: ${A2A_AGENT_CLIENT_PROVIDER}
A2A_AGENT_CLIENT_MODEL: ${A2A_AGENT_CLIENT_MODEL}
A2A_AGENT_CLIENT_API_KEY: ""
A2A_AGENT_CLIENT_BASE_URL: http://inference-gateway:8080/v1
A2A_AGENT_CLIENT_TIMEOUT: 30s
Expand All @@ -72,13 +71,14 @@ services:
image: ghcr.io/inference-gateway/inference-gateway:latest
container_name: inference-gateway
environment:
DEEPSEEK_API_KEY: ${DEEPSEEK_API_KEY}
GOOGLE_API_KEY: ${GOOGLE_API_KEY}
ENVIRONMENT: development
SERVER_READ_TIMEOUT: 530s
SERVER_WRITE_TIMEOUT: 530s
CLIENT_TIMEOUT: 530s
CLIENT_IDLE_CONN_TIMEOUT: 30s
env_file:
.env.gateway
CLIENT_IDLE_CONN_TIMEOUT: 130s
CLIENT_RESPONSE_HEADER_TIMEOUT: 120s
networks:
- a2a-network

Expand All @@ -89,23 +89,10 @@ services:
INFER_LOGGING_DEBUG: true
INFER_GATEWAY_URL: http://inference-gateway:8080
INFER_A2A_ENABLED: true
INFER_TOOLS_ENABLED: true
INFER_TOOLS_QUERY_ENABLED: true
INFER_TOOLS_TASK_ENABLED: true
INFER_TOOLS_BASH_ENABLED: false
INFER_TOOLS_TODO_WRITE_ENABLED: false
INFER_TOOLS_WRITE_ENABLED: false
INFER_TOOLS_READ_ENABLED: false
INFER_TOOLS_DELETE_ENABLED: false
INFER_TOOLS_EDIT_ENABLED: false
INFER_TOOLS_GREP_ENABLED: false
INFER_TOOLS_TREE_ENABLED: false
INFER_TOOLS_WEB_FETCH_ENABLED: false
INFER_TOOLS_WEB_SEARCH_ENABLED: false
INFER_TOOLS_GITHUB_ENABLED: false
INFER_TOOLS_ENABLED: false
INFER_AGENT_MODEL: deepseek/deepseek-chat
INFER_A2A_AGENTS: 'http://agent:8080'
INFER_A2A_CACHE_ENABLED: false
INFER_A2A_AGENTS: |
http://agent:8080
command:
- chat
networks:
Expand Down
76 changes: 76 additions & 0 deletions example/integration_demo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# CSV Export Integration Demo

This document demonstrates how to use the new `write_to_csv` skill in combination with the existing `extract_data` skill for complete data collection workflows.

## Workflow Example

1. **Navigate to a webpage**:
```json
{
"skill": "navigate_to_url",
"args": {
"url": "https://example.com/products"
}
}
```

2. **Extract data from the page**:
```json
{
"skill": "extract_data",
"args": {
"extractors": [
{
"name": "product_name",
"selector": ".product-title",
"multiple": true
},
{
"name": "price",
"selector": ".product-price",
"multiple": true
},
{
"name": "rating",
"selector": ".product-rating",
"attribute": "data-rating",
"multiple": true
}
],
"format": "json"
}
}
```

3. **Write the extracted data to CSV**:
```json
{
"skill": "write_to_csv",
"args": {
"data": [
{"product_name": "Product A", "price": "$29.99", "rating": "4.5"},
{"product_name": "Product B", "price": "$39.99", "rating": "4.2"},
{"product_name": "Product C", "price": "$19.99", "rating": "4.8"}
],
"file_path": "/tmp/products.csv",
"headers": ["product_name", "price", "rating"],
"include_headers": true
}
}
```

## Features Supported

- **Custom Headers**: Specify column order and names
- **Append Mode**: Add to existing CSV files without overwriting
- **Flexible Data**: Handles arrays, objects, and primitive values
- **Error Handling**: Validates data format and file operations
- **Directory Creation**: Automatically creates parent directories

## Use Cases

- **E-commerce Data Collection**: Extract product information, prices, and reviews
- **News Aggregation**: Collect headlines, dates, and article links
- **Financial Data**: Gather stock prices, market data, and trading volumes
- **Contact Information**: Extract business details from directory sites
- **Event Listings**: Collect event names, dates, venues, and prices
2 changes: 1 addition & 1 deletion internal/logger/logger.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading