Skip to content
BranchesTags

Repository files navigation

Azure DevOps Artifact Scraper

A Python application that searches all Azure DevOps artefacts for specified keywords and returns results sorted by type.

Features

  • Concurrent Search: Searches multiple artifact types simultaneously for optimal performance
  • Progress Tracking: Keeps users informed with real-time progress updates
  • Case-Sensitive Search: Optional flag to enable exact case matching for keywords
  • Whole Word Matching: Optional flag to match complete words only (avoid partial matches)
  • Line Number Tracking: Shows exact line numbers where keywords are found in files and text fields
  • Author Information: Displays author/creator information for work items and file commits
  • Debug Mode: Detailed logging for troubleshooting and monitoring search progress
  • Branch Control: Option to search all branches or just default branches for faster searches
  • Timing Information: Track and display scan start time, end time, and total duration
  • URL Encoding: Properly handles spaces and special characters in artifact names
  • Markdown Report Generation: Automatically saves results to timestamped markdown files
  • Comprehensive Coverage: Searches across:
    • Git repositories (file names and content across all branches)
    • Work items (all fields including custom fields)
    • Commit history (messages, author names, and author emails across all branches)
    • Work item update history
    • Pipelines (build definitions)
    • Build artefacts
    • Project wikis (including code wikis across all branches)
  • Sorted Results: Returns results organized by artifact type with direct links
  • Flexible Input: Accepts Azure DevOps project name and PAT token as inputs
  • Keyword File Support: Reads search keywords from a configurable file
  • Batch Processing: Efficiently handles large result sets with automatic batching

Installation

  1. Install Python 3.8 or higher

  2. Install required dependencies:

pip install -r requirements.txt

Configuration

  1. Create or edit the keywords.txt file with your search keywords (one per line). A sample keyterms.txt file is included in the repository if you prefer to start from that template:
authentication
security
database
api
  1. Generate a Personal Access Token (PAT) in Azure DevOps:
    • Go to User Settings → Personal access tokens
    • Create a new token with at least these scopes:
      • Code: Read
      • Work Items: Read
      • Build: Read
      • Project and Team: Read

Usage

Command Line Arguments

python azdo_scraper.py <organization_url> <project> <pat> [keywords_file] [case_sensitive] [debug] [all_branches] [whole_word_only]

Parameters:

  • organization_url: Your Azure DevOps organization URL (e.g., https://dev.azure.com/myorg)
  • project: Project name
  • pat: Personal Access Token
  • keywords_file: (Optional) Path to keywords file, defaults to keywords.txt
  • case_sensitive: (Optional) Enable case-sensitive search (true or false, defaults to false)
  • debug: (Optional) Enable detailed debug logging (true or false, defaults to false)
  • all_branches: (Optional) Search all branches instead of just default branch (true or false, defaults to false)
  • whole_word_only: (Optional) Match whole words only, ignore keywords within other words (true or false, defaults to false)

Examples:

Case-insensitive search with default branch only (fastest):

python azdo_scraper.py https://dev.azure.com/myorg MyProject your_token keywords.txt

Use a custom keywords file:

python azdo_scraper.py https://dev.azure.com/myorg MyProject your_token my-keywords.txt

Case-sensitive search with debug logging:

python azdo_scraper.py https://dev.azure.com/myorg MyProject your_token keywords.txt true true

Search all branches (slower but more thorough):

python azdo_scraper.py https://dev.azure.com/myorg MyProject your_token keywords.txt false false true

Match whole words only (avoid partial matches in larger words):

python azdo_scraper.py https://dev.azure.com/myorg MyProject your_token keywords.txt false false false true

Using Make

You can also use the Makefile for easier execution:

# Basic search (case-insensitive, default branch only)
make AZDO_ORG=https://dev.azure.com/myorg AZDO_PROJECT=MyProject AZDO_PAT=your_token run

# Use a custom keywords file
make AZDO_ORG=https://dev.azure.com/myorg AZDO_PROJECT=MyProject AZDO_PAT=your_token KEYWORDS=my-keywords.txt run

# Case-sensitive search
make AZDO_ORG=https://dev.azure.com/myorg AZDO_PROJECT=MyProject AZDO_PAT=your_token CASE_SENSITIVE=true run

# Enable debug logging for detailed progress
make AZDO_ORG=https://dev.azure.com/myorg AZDO_PROJECT=MyProject AZDO_PAT=your_token DEBUG=true run

# Search all branches (more comprehensive but slower)
make AZDO_ORG=https://dev.azure.com/myorg AZDO_PROJECT=MyProject AZDO_PAT=your_token ALL_BRANCHES=true run

# Combine multiple flags
make AZDO_ORG=https://dev.azure.com/myorg AZDO_PROJECT=MyProject AZDO_PAT=your_token \
  KEYWORDS=custom.txt CASE_SENSITIVE=true DEBUG=true ALL_BRANCHES=false WHOLE_WORD_ONLY=true run

Interactive Mode

Run without arguments to enter details interactively:

python azdo_scraper.py

You'll be prompted for all parameters including:

  • Organization URL
  • Project name
  • Personal Access Token
  • Keywords file path
  • Case-sensitive search option
  • Debug logging option
  • All branches search option
  • Whole word only search option

Output

The application will:

  1. Show progress updates as it searches each artifact type
  2. Display timing information (scan start, end, and total duration)
  3. Show detailed file-by-file progress when DEBUG mode is enabled
  4. Display a summary of results in the console organized by type
  5. Automatically save results to a timestamped markdown file (e.g., azdo_search_results_20251028_132020.md)
  6. Include direct links, matching keywords, and metadata for each result

Console Output

Example console output:

================================================================================
SEARCH RESULTS
================================================================================

⏱️  Scan Start Time: 2025-11-10 09:12:01
⏱️  Scan End Time:   2025-11-10 09:13:44
⏱️  Total Duration:  0:01:43

Total results found: 6

--------------------------------------------------------------------------------
REPOSITORYFILE (2 results)
--------------------------------------------------------------------------------

  📄 ServiceConfig: src/settings.py
  🔗 Direct Link: https://dev.azure.com/myorg/MyProject/_git/ServiceConfig?path=/src/settings.py
  🔍 Keywords: authentication, token
  📍 Lines: 42, 99
  👤 Author: Alexis Rivers
  ℹ️  Details: repository: ServiceConfig, branch: main

--------------------------------------------------------------------------------
WORKITEM (2 results)
--------------------------------------------------------------------------------

  📄 User Story #1234: Refresh token reliability
  🔗 Direct Link: https://dev.azure.com/myorg/MyProject/_workitems/edit/1234
  🔍 Keywords: authentication
  📝 Matches: 'Ensure refresh tokens stay valid', 'Validate authentication retry flow'
  👤 Author: Myria Chen
  ℹ️  Details: type: User Story, state: Active, assigned_to: John Doe

--------------------------------------------------------------------------------
WIKIPAGE (1 results)
--------------------------------------------------------------------------------

  📄 Security Guidelines: Token usage
  🔗 Direct Link: https://dev.azure.com/myorg/MyProject/_wiki/wikis/Docs?pagePath=%2FToken%20usage
  🔍 Keywords: token
  📝 Match: 'Token rotation policy requires 24h expiration'
  👤 Author: Documentation Team

--------------------------------------------------------------------------------
COMMIT (1 results)
--------------------------------------------------------------------------------

  📄 Commit in ServiceConfig (main): Improve token refresh flow
  🔗 Direct Link: https://dev.azure.com/myorg/MyProject/_git/ServiceConfig/commit/a1b2c3d4
  🔍 Keywords: authentication
  ℹ️  Details: repository: ServiceConfig, branch: main, commit_id: a1b2c3d4, author: Alexis Rivers, author_email: alexis.rivers@example.com

Markdown Report

Each search automatically generates a markdown report with:

  • Search metadata (timestamp, organization, project, keywords, case-sensitivity, branch mode)
  • Timing information (scan start time, end time, total duration)
  • Table of contents with result counts by type
  • Detailed sections for each artifact type with clickable links
  • Complete metadata for each result

Report filename format: azdo_search_results_YYYYMMDD_HHMMSS.md

Example report structure:

# Azure DevOps Search Results

**Generated:** 2025-10-28 13:20:20
**Organization:** https://dev.azure.com/myorg
**Project:** MyProject
**Keywords:** authentication, api, database
**Case Sensitive:** No
**All Branches:** No (default branch only)
**Scan Start Time:** 2025-10-28 13:20:15
**Scan End Time:** 2025-10-28 13:22:48
**Total Scan Duration:** 0:02:33
**Total Results:** 42

## Table of Contents
- [Repository](#repository) (5 results)
- [Workitem](#workitem) (15 results)
...

Configuration Options

DEBUG Mode

Enable DEBUG mode for detailed file-by-file progress:

make AZDO_ORG=... AZDO_PROJECT=... AZDO_PAT=... DEBUG=true run

Debug mode shows:

  • Individual files being checked in repositories
  • Wiki pages being processed
  • Detailed error messages for troubleshooting
  • Timing information for each search phase

ALL_BRANCHES Mode

By default, the scraper only searches the default branch (e.g., main or master) for faster performance. Enable ALL_BRANCHES to search all branches:

make AZDO_ORG=... AZDO_PROJECT=... AZDO_PAT=... ALL_BRANCHES=true run

Considerations:

  • Default (ALL_BRANCHES=false): Faster, searches only default branch
  • ALL_BRANCHES=true: Slower but more comprehensive, searches all branches
    • Repository files: Searches all branches for file content
    • Commit history: 1000 commits per branch (vs 100 for default branch only)
    • Code wikis: Searches all wiki branches
  • Recommended: Use default mode unless you need to find keywords in feature branches

WHOLE_WORD_ONLY Mode

By default, the scraper finds keywords anywhere within text (substring matching). Enable WHOLE_WORD_ONLY to match only complete words:

make AZDO_ORG=... AZDO_PROJECT=... AZDO_PAT=... WHOLE_WORD_ONLY=true run

Examples of matching behavior:

Keyword Text Default Match Whole Word Match
test "This is a test" ✅ Yes ✅ Yes
test "Testing phase" ✅ Yes ❌ No
test "contest results" ✅ Yes ❌ No
test "test-driven" ✅ Yes ✅ Yes
test "test_function" ✅ Yes ❌ No

When to use:

  • Default (WHOLE_WORD_ONLY=false): Find all occurrences, including within larger words
  • WHOLE_WORD_ONLY=true: Avoid false positives from partial matches within compound words or identifiers

Performance

The application uses asynchronous operations to search multiple artifact types concurrently, making it highly performant even with large projects and many keywords.

Performance Tips

  • Use default branch mode (ALL_BRANCHES=false) for faster searches
  • Limit keywords to the most relevant terms
  • Use DEBUG mode only when troubleshooting
  • The application automatically batches large work item queries for optimal API usage

Recent Improvements

Azure DevOps Search API Integration (Optional)

  • Automatic detection and use of Azure DevOps Search API when available
  • Dramatically faster repository searches through pre-indexed content
  • Falls back to direct file search if Search API is not enabled
  • Search API benefits:
    • Instant results from pre-built index (no file downloads)
    • Searches ALL file types (not just text files)
    • Handles large codebases efficiently
    • Reduces API calls by 100x-1000x
  • Progress messages indicate which search method is being used
  • No configuration needed - automatically detected and used

Branch Support (ALL_BRANCHES)

  • Comprehensive branch support for all Git-based searches
  • Repository file search: Find keywords in files across all branches
  • Commit history search: Search commits across all branches with 1000 commits per branch (10x increase)
  • Code wiki search: Search wiki content across all branches
  • Branch names included in results for better context
  • Default mode searches only default branch for optimal performance

Enhanced Commit Search

  • Now searches commit author names and emails in addition to commit messages
  • Find all commits by a specific person or email domain
  • Track commits from contractors, partners, or specific organizations
  • Author email included in commit result details
  • Use cases:
    • Search for "john.doe@company.com" to find all commits by that author
    • Search for "contractor.com" to track commits from contractors
    • Search for person names to find their contributions

URL Encoding

All generated URLs now properly encode spaces and special characters, ensuring clickable links work correctly even when:

  • Project names contain spaces (e.g., "My Project")
  • Repository names contain spaces (e.g., "My Repo")
  • File paths contain spaces (e.g., "/folder/my file.txt")
  • Wiki names or page paths contain spaces

Whole Word Keyword Matching

  • New WHOLE_WORD_ONLY flag prevents false positives from partial matches in identifiers or concatenated words
  • Works alongside case-sensitive and case-insensitive searches
  • Supported via CLI argument, interactive mode prompt, and Makefile flag (WHOLE_WORD_ONLY=true)
  • Matching summaries now highlight the exact lines of work items and wiki pages where whole-word hits occur

Wiki Search

  • Fixed project wiki search to recursively collect all pages (not just root page)
  • Improved URL generation using WikiPage API attributes
  • Cleaner debug output matching other search sections

Timing Information

  • Track and display scan start time, end time, and total duration
  • Timing information included in both console output and markdown reports
  • Helps identify performance bottlenecks

Development

Running Tests

The project includes comprehensive unit tests covering all major functionality:

# Run tests with coverage
make test

# Run tests with detailed HTML coverage report
make test-verbose

# Run tests directly with pytest
pytest test_azdo_scraper.py -v

Test Coverage: Extensive unit tests covering:

  • SearchResult dataclass creation
  • Searcher initialization and configuration
  • Case-sensitive and case-insensitive keyword matching
  • Search API availability detection
  • Search API integration with fallback
  • Repository, commit, work item, pipeline, build, and wiki searches
  • Result deduplication and sorting
  • Keyword loading from files
  • Progress callback functionality
  • Complete search orchestration

Code Quality

# Format code with black (120 char line length)
make format

# Check code formatting
make format-check

# Run linting (flake8 + pylint)
make lint

# Run all checks (format check + linting)
make check

Code Quality Standards:

  • Line length: 120 characters
  • Linting: flake8 and pylint
  • Formatting: black
  • Current rating: 9.07/10 (pylint)

Development Dependencies

Install development dependencies for testing and code quality:

make install-dev

This installs:

  • pytest - Testing framework
  • pytest-asyncio - Async test support
  • pytest-cov - Coverage reporting
  • black - Code formatter
  • flake8 - Linting
  • pylint - Code analysis

Limitations

  • Build search is limited to the 100 most recent builds for performance
  • Commit search limits:
    • Default mode: 100 commits from default branch
    • ALL_BRANCHES mode: 1000 commits per branch
  • Azure DevOps Search API (optional):
    • Provides much faster repository searches when enabled
    • Requires Azure DevOps Search license (paid feature)
    • Automatically falls back to direct file search if not available
  • Direct file search (when Search API not available):
    • Slower for large repositories (downloads each file)
    • Limited to specific file extensions (.txt, .md, .py, .js, etc.)
  • The free version of Azure DevOps may have API rate limits
  • Generated markdown reports are saved locally and excluded from version control (via .gitignore)

Troubleshooting

  • Authentication Error: Ensure your PAT token has the correct permissions
  • Project Not Found: Verify the project name and organization URL are correct
  • No Results: Check that keywords match content in your Azure DevOps project
  • API Rate Limiting: If you encounter rate limits, try reducing the number of keywords or implementing delays
  • Markdown Report Not Found: Check the current directory for files matching azdo_search_results_*.md

License

MIT License

About

Searches all Azure DevOps artifacts for specified keywords

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages