Feature Request: Add Element Locator Tool for aria-ref to ref Mapping

<html>
<body>
<html><head></head><body><h1>Feature Request: Add Element Locator Tool for aria-ref to ref Mapping</h1>
<h2>🎯 Feature Summary</h2>
<p>Add a new tool that allows querying elements using <strong>aria-ref</strong> attributes within captured PageSnapshots and returns the corresponding MCP internal <code>ref</code> identifier along with traditional CSS and XPath selectors for enhanced automation workflows.</p>
<h2>🔍 Background &amp; Context</h2>
<p>Currently, the Playwright MCP workflow requires:</p>
<ol>
<li>Call <code>browser_snapshot</code> to get accessibility snapshot</li>
<li>Manually identify the target element's <code>ref</code> from the snapshot</li>
<li>Use the <code>ref</code> parameter in interaction tools like <code>browser_click</code>, <code>browser_type</code>, etc.</li>
</ol>
<p>The server enables LLMs to interact with web pages through structured accessibility snapshots, and all interaction tools require a <code>ref</code> parameter which is the "Exact target element reference from the page snapshot".</p>
<p>However, there's currently no programmatic way to locate elements by HTML attributes (like <code>aria-ref</code>) and retrieve their corresponding MCP <code>ref</code> identifier.</p>
<h2>🛠️ Proposed Tool</h2>
<h3>Tool Name: <code>browser_find_element</code></h3>
<h3>Description</h3>
<p>Locate elements within the current PageSnapshot using various attribute-based selectors and return their MCP <code>ref</code> identifiers along with standard CSS/XPath selectors.</p>
<h3>Parameters</h3>
<pre><code class="language-typescript">{
  // Query parameters (at least one required)
  "aria_ref"?: string,           // aria-ref attribute value
  "data_testid"?: string,        // data-testid attribute value  
  "id"?: string,                 // HTML id attribute
  "css_selector"?: string,       // CSS selector
  
  // Options
  "include_selectors"?: boolean, // Include CSS/XPath (default: true)
  "multiple"?: boolean          // Return all matches vs first match (default: false)
}
</code></pre>
<h3>Expected Response</h3>
<pre><code class="language-json">{
  "found": true,
  "elements": [
    {
      "ref": "node-123",  // MCP internal reference for use with other tools
      "element_info": {
        "role": "button",
        "name": "Submit Form", 
        "tag": "button",
        "attributes": {
          "aria-ref": "submit-btn",
          "class": "btn btn-primary"
        }
      },
      "selectors": {
        "css_path": "form.main-form button[aria-ref='submit-btn']",
        "xpath": "//form[@class='main-form']//button[@aria-ref='submit-btn']",
        "css_nth": "form.main-form button:nth-child(2)"
      }
    }
  ],
  "total_found": 1
}
</code></pre>
<h2>🎯 Use Cases &amp; Integration</h2>
<h3>1. <strong>Streamlined Automation Workflow</strong></h3>
<pre><code class="language-javascript">// Before (current workflow)
const snapshot = await browser_snapshot();
// Manual inspection needed to find ref for aria-ref="submit"
const clickResult = await browser_click({
  element: "Submit button",
  ref: "node-123" // Had to manually identify this
});

// After (with new tool)
const findResult = await browser_find_element({
  aria_ref: "submit"
});
const clickResult = await browser_click({
  element: "Submit button", 
  ref: findResult.elements[0].ref // Programmatically obtained
});
</code></pre>
<h3>2. <strong>Enhanced Test Automation</strong></h3>
<ul>
<li>Generate stable selectors for CI/CD pipelines</li>
<li>Bridge gap between accessibility-first development and automation</li>
<li>Support for multiple selector strategies in one tool</li>
</ul>
<h3>3. <strong>Debugging &amp; Development Support</strong></h3>
<ul>
<li>Quick element identification during development</li>
<li>Validation of accessibility attributes</li>
<li>Cross-reference between MCP refs and standard selectors</li>
</ul>
<h2>🏗️ Technical Implementation Considerations</h2>
<h3>Integration with Existing Architecture</h3>
<ul>
<li><strong>Leverage Current PageSnapshot</strong>: Build upon existing <code>browser_snapshot</code> infrastructure</li>
<li><strong>Maintain Consistency</strong>: Follow same response patterns as other MCP tools</li>
<li><strong>Performance</strong>: Cache snapshot data to avoid redundant accessibility tree traversals</li>
</ul>
<h3>Selector Generation Strategy</h3>
<p>Based on Playwright's accessibility tree structure that contains information about elements, their roles, names, values, and relationships:</p>
<pre><code class="language-javascript">function generateSelectors(element) {
  return {
    css_path: buildCSSPath(element),     // Hierarchical CSS selector
    xpath: buildXPath(element),          // XPath expression
    css_nth: buildNthChildCSS(element)   // Nth-child based selector
  };
}
</code></pre>
<h3>Error Handling</h3>
<ul>
<li>Handle cases where attributes are not found</li>
<li>Support partial matching for dynamic content</li>
<li>Graceful degradation when selector generation fails</li>
</ul>
<h2>🔄 Compatibility with Current MCP Tools</h2>
<p>The new tool integrates seamlessly with the existing tool ecosystem:</p>

Current Tool | Integration Benefit
-- | --
browser_snapshot | Uses same accessibility data source
browser_click | Provides ref parameter directly
browser_type | Enables attribute-based form field targeting
browser_hover | Supports accessible element hover actions
browser_drag | Facilitates drag-and-drop with accessible elements


<h2>📋 Acceptance Criteria</h2>
<ul>
<li>[ ] Successfully locate elements by <code>aria-ref</code>, <code>data-testid</code>, <code>id</code>, and CSS selectors</li>
<li>[ ] Return valid MCP <code>ref</code> identifiers compatible with existing interaction tools</li>
<li>[ ] Generate stable and unique CSS Path and XPath selectors</li>
<li>[ ] Handle multiple matches with configurable return behavior</li>
<li>[ ] Maintain performance with large PageSnapshots (&gt;1000 elements)</li>
<li>[ ] Provide comprehensive error messages for debugging</li>
<li>[ ] Include accessibility metadata in responses</li>
<li>[ ] Follow existing MCP tool parameter and response patterns</li>
</ul>
<h2>🚀 Future Enhancement Opportunities</h2>
<h3>Phase 2 Features</h3>
<ul>
<li><strong>Fuzzy Matching</strong>: Support approximate text matching for accessible names</li>
<li><strong>Selector Validation</strong>: Verify generated selectors against live page state</li>
<li><strong>Batch Operations</strong>: Find multiple elements in single tool call</li>
<li><strong>Performance Metrics</strong>: Include selector generation timing in responses</li>
</ul>
<h3>Integration Possibilities</h3>
<ul>
<li><strong>Test Generation</strong>: Auto-generate Playwright test code using returned selectors</li>
<li><strong>Page Object Models</strong>: Support structured element mapping for large applications</li>
<li><strong>Accessibility Auditing</strong>: Identify elements missing required accessibility attributes</li>
</ul>
<h2>📖 Technical References</h2>
<ul>
<li><a href="https://playwright.dev/docs/accessibility-testing">Playwright Accessibility Tree API</a></li>
<li><a href="https://www.w3.org/WAI/ARIA/">ARIA Reference Specification</a></li>
<li>Playwright ARIA Snapshots: YAML representation of the accessibility tree</li>
<li><a href="https://github.com/modelcontextprotocol/specification">MCP Tool Development Guidelines</a></li>
</ul>
<hr>
<p><strong>Priority</strong>: High<br>
<strong>Complexity</strong>: Medium<br>
<strong>Dependencies</strong>: Existing PageSnapshot infrastructure<br>
<strong>Estimated Effort</strong>: 2-3 development cycles</p>
<p><strong>Note</strong>: This tool addresses the gap between accessibility-first development practices and automation needs, making MCP more powerful for developers using semantic HTML attributes.</p></body></html>
</body>
</html># Feature Request: Add Element Locator Tool for aria-ref to ref Mapping

## 🎯 Feature Summary

Add a new tool that allows querying elements using **aria-ref** attributes within captured PageSnapshots and returns the corresponding MCP internal `ref` identifier along with traditional CSS and XPath selectors for enhanced automation workflows.

## 🔍 Background & Context

Currently, the Playwright MCP workflow requires:
1. Call `browser_snapshot` to get accessibility snapshot
2. Manually identify the target element's `ref` from the snapshot 
3. Use the `ref` parameter in interaction tools like `browser_click`, `browser_type`, etc.

The server enables LLMs to interact with web pages through structured accessibility snapshots, and all interaction tools require a `ref` parameter which is the "Exact target element reference from the page snapshot".

However, there's currently no programmatic way to locate elements by HTML attributes (like `aria-ref`) and retrieve their corresponding MCP `ref` identifier.

## 🛠️ Proposed Tool

### Tool Name: `browser_find_element`

### Description
Locate elements within the current PageSnapshot using various attribute-based selectors and return their MCP `ref` identifiers along with standard CSS/XPath selectors.

### Parameters
```typescript
{
  // Query parameters (at least one required)
  "aria_ref"?: string,           // aria-ref attribute value
  "data_testid"?: string,        // data-testid attribute value  
  "id"?: string,                 // HTML id attribute
  "css_selector"?: string,       // CSS selector
  
  // Options
  "include_selectors"?: boolean, // Include CSS/XPath (default: true)
  "multiple"?: boolean          // Return all matches vs first match (default: false)
}
```

### Expected Response
```json
{
  "found": true,
  "elements": [
    {
      "ref": "node-123",  // MCP internal reference for use with other tools
      "element_info": {
        "role": "button",
        "name": "Submit Form", 
        "tag": "button",
        "attributes": {
          "aria-ref": "submit-btn",
          "class": "btn btn-primary"
        }
      },
      "selectors": {
        "css_path": "form.main-form button[aria-ref='submit-btn']",
        "xpath": "//form[@class='main-form']//button[@aria-ref='submit-btn']",
        "css_nth": "form.main-form button:nth-child(2)"
      }
    }
  ],
  "total_found": 1
}
```

## 🎯 Use Cases & Integration

### 1. **Streamlined Automation Workflow**
```javascript
// Before (current workflow)
const snapshot = await browser_snapshot();
// Manual inspection needed to find ref for aria-ref="submit"
const clickResult = await browser_click({
  element: "Submit button",
  ref: "node-123" // Had to manually identify this
});

// After (with new tool)
const findResult = await browser_find_element({
  aria_ref: "submit"
});
const clickResult = await browser_click({
  element: "Submit button", 
  ref: findResult.elements[0].ref // Programmatically obtained
});
```

### 2. **Enhanced Test Automation**
- Generate stable selectors for CI/CD pipelines
- Bridge gap between accessibility-first development and automation
- Support for multiple selector strategies in one tool

### 3. **Debugging & Development Support**
- Quick element identification during development
- Validation of accessibility attributes
- Cross-reference between MCP refs and standard selectors

## 🏗️ Technical Implementation Considerations

### Integration with Existing Architecture
- **Leverage Current PageSnapshot**: Build upon existing `browser_snapshot` infrastructure
- **Maintain Consistency**: Follow same response patterns as other MCP tools
- **Performance**: Cache snapshot data to avoid redundant accessibility tree traversals

### Selector Generation Strategy
Based on Playwright's accessibility tree structure that contains information about elements, their roles, names, values, and relationships:

```javascript
function generateSelectors(element) {
  return {
    css_path: buildCSSPath(element),     // Hierarchical CSS selector
    xpath: buildXPath(element),          // XPath expression
    css_nth: buildNthChildCSS(element)   // Nth-child based selector
  };
}
```

### Error Handling
- Handle cases where attributes are not found
- Support partial matching for dynamic content
- Graceful degradation when selector generation fails

## 🔄 Compatibility with Current MCP Tools

The new tool integrates seamlessly with the existing tool ecosystem:

| Current Tool | Integration Benefit |
|--------------|-------------------|
| `browser_snapshot` | Uses same accessibility data source |
| `browser_click` | Provides `ref` parameter directly |
| `browser_type` | Enables attribute-based form field targeting |
| `browser_hover` | Supports accessible element hover actions |
| `browser_drag` | Facilitates drag-and-drop with accessible elements |

## 📋 Acceptance Criteria

- [ ] Successfully locate elements by `aria-ref`, `data-testid`, `id`, and CSS selectors
- [ ] Return valid MCP `ref` identifiers compatible with existing interaction tools
- [ ] Generate stable and unique CSS Path and XPath selectors
- [ ] Handle multiple matches with configurable return behavior
- [ ] Maintain performance with large PageSnapshots (>1000 elements)
- [ ] Provide comprehensive error messages for debugging
- [ ] Include accessibility metadata in responses
- [ ] Follow existing MCP tool parameter and response patterns

## 🚀 Future Enhancement Opportunities

### Phase 2 Features
- **Fuzzy Matching**: Support approximate text matching for accessible names
- **Selector Validation**: Verify generated selectors against live page state  
- **Batch Operations**: Find multiple elements in single tool call
- **Performance Metrics**: Include selector generation timing in responses

### Integration Possibilities
- **Test Generation**: Auto-generate Playwright test code using returned selectors
- **Page Object Models**: Support structured element mapping for large applications
- **Accessibility Auditing**: Identify elements missing required accessibility attributes

## 📖 Technical References

- [[Playwright Accessibility Tree API](https://playwright.dev/docs/accessibility-testing)](https://playwright.dev/docs/accessibility-testing)
- [[ARIA Reference Specification](https://www.w3.org/WAI/ARIA/)](https://www.w3.org/WAI/ARIA/)
- Playwright ARIA Snapshots: YAML representation of the accessibility tree
- [[MCP Tool Development Guidelines](https://github.com/modelcontextprotocol/specification)](https://github.com/modelcontextprotocol/specification)

---

**Priority**: High  
**Complexity**: Medium  
**Dependencies**: Existing PageSnapshot infrastructure  
**Estimated Effort**: 2-3 development cycles

**Note**: This tool addresses the gap between accessibility-first development practices and automation needs, making MCP more powerful for developers using semantic HTML attributes.

Current Tool	Integration Benefit
browser_snapshot	Uses same accessibility data source
browser_click	Provides ref parameter directly
browser_type	Enables attribute-based form field targeting
browser_hover	Supports accessible element hover actions
browser_drag	Facilitates drag-and-drop with accessible elements

Current Tool	Integration Benefit
`browser_snapshot`	Uses same accessibility data source
`browser_click`	Provides `ref` parameter directly
`browser_type`	Enables attribute-based form field targeting
`browser_hover`	Supports accessible element hover actions
`browser_drag`	Facilitates drag-and-drop with accessible elements

Feature Request: Add Element Locator Tool for aria-ref to ref Mapping #588

Description

Feature Request: Add Element Locator Tool for aria-ref to ref Mapping

🎯 Feature Summary

🔍 Background & Context

🛠️ Proposed Tool

Tool Name: browser_find_element

Description

Parameters

Expected Response

🎯 Use Cases & Integration

1. Streamlined Automation Workflow

2. Enhanced Test Automation

3. Debugging & Development Support

🏗️ Technical Implementation Considerations

Integration with Existing Architecture

Selector Generation Strategy

Error Handling

🔄 Compatibility with Current MCP Tools

📋 Acceptance Criteria

🚀 Future Enhancement Opportunities

Phase 2 Features

Integration Possibilities

📖 Technical References

🎯 Feature Summary

🔍 Background & Context

🛠️ Proposed Tool

Tool Name: browser_find_element

Description

Parameters

Expected Response

🎯 Use Cases & Integration

1. Streamlined Automation Workflow

2. Enhanced Test Automation

3. Debugging & Development Support

🏗️ Technical Implementation Considerations

Integration with Existing Architecture

Selector Generation Strategy

Error Handling

🔄 Compatibility with Current MCP Tools

📋 Acceptance Criteria

🚀 Future Enhancement Opportunities

Phase 2 Features

Integration Possibilities

📖 Technical References

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Tool Name: `browser_find_element`

Tool Name: `browser_find_element`