PanoramaBridge

A Python Qt6 application for monitoring local directories and automatically transferring files to Panorama WebDAV servers. This tool provides intelligent file monitoring with secure credential storage, robust upload capabilities, and advanced features specifically designed for mass spectrometer workflows.

Quick Links

Installation - Get started with pip or from source
User Guide - How to use the application
Windows Build - Create standalone executable
GitHub Actions - Automated builds and releases
Technical Docs - Architecture and implementation details
AI Agents Guide - Guide for AI-assisted development

Features

Core Features

Intelligent File Monitoring: OS-level file system events with optional backup polling
WebDAV Integration: Seamless integration with Panorama WebDAV servers
Secure Credential Storage: Uses system keyring for safe password storage
Chunked Upload: Efficient handling of large files with progress tracking
Directory Structure Preservation: Maintains folder hierarchy on remote server
Remote Directory Browser: Navigate and create folders on the WebDAV server

Advanced Features

Smart Locked File Handling: Automatically detects and retries locked files from mass spectrometers
Checksum Caching: Dramatic performance improvements (up to 1700x faster for unchanged files)
Upload History Tracking: Persistent tracking of successfully uploaded files with integrity verification
Remote File Integrity Verification: Comprehensive verification system that ensures all local files are properly uploaded to the remote server, with intelligent conflict resolution for file differences
Progress Indication: Real-time progress bars with elapsed time and countdown timers
Windows Native Support: Optimized .venv-win virtual environment for better file system event detection
Configurable Retry Logic: Customizable wait times and retry attempts for locked files

Performance & Reliability

OS Event-Driven Monitoring: Immediate file detection without polling overhead
Intelligent Conflict Resolution: Smart file comparison and conflict handling with user choice
Remote Integrity Verification: Automatically verifies and resolves missing or changed remote files
Persistent State Management: Maintains upload history across application restarts
Comprehensive Logging: Detailed logs with menu-based access for troubleshooting
Cross-platform: Works on Windows, Linux, and macOS (Windows native recommended)

Requirements

Python 3.9 or later
Operating System: Windows (recommended), Linux, or macOS

Python Dependencies

PyQt6>=6.4.0
watchdog>=3.0.0
requests>=2.31.0
keyring>=24.0.0
keyrings.alt>=5.0.0

Installation

Prerequisites: Install Python

PanoramaBridge requires Python 3.9 or later. If you don't have Python installed:

Windows:

Download Python from python.org/downloads
Run the installer
Important: Check the box "Add Python to PATH" during installation
Click "Install Now"
Verify installation by opening Command Prompt or PowerShell and running:
```
python --version
```

macOS:

# Using Homebrew (recommended)
brew install python

# Verify installation
python3 --version

Linux (Ubuntu/Debian):

sudo apt update
sudo apt install python3 python3-pip python3-venv

# Verify installation
python3 --version

Option 1: Install from PyPI (Recommended)

Once Python is installed, open a terminal (Command Prompt, PowerShell, or Terminal) and run:

pip install panoramabridge

Then run the application:

panoramabridge

Note: On some systems you may need to use pip3 instead of pip, and python3 instead of python.

Option 2: Install from Source

Clone the repository:

git clone https://github.com/maccoss/PanoramaBridge.git
cd PanoramaBridge

Create virtual environment (recommended):

Windows (Native Performance):

python -m venv .venv-win
.venv-win\Scripts\activate

Linux/macOS:

python -m venv .venv
source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Run the application:

python panoramabridge.py

Option 3: Download Windows Executable

Download a pre-built executable from the Releases page:

Download PanoramaBridge.exe
Run the executable (Windows may show a security warning - click "More info" → "Run anyway")
No Python installation required

Quick Start

Launch the application: python panoramabridge.py
Configure WebDAV connection:
- Go to "Remote Settings" tab
- Enter your Panorama server URL (e.g., https://panoramaweb.org)
- Add your username and password
- Click "Test Connection" to verify
Set up monitoring:
- Go to "Local Monitoring" tab
- Select directory to monitor
- Specify file extensions (e.g., raw, sld, csv)
- Choose remote destination folder
Start monitoring: Click "Start Monitoring"
View progress: Check "Transfer Status" tab for real-time updates

Creating a Windows Executable (Optional)

For detailed Windows build instructions, see BUILD_WINDOWS.md

To create a standalone .exe file:

1. Install PyInstaller

pip install pyinstaller

2. Create the Executable

pyinstaller --onefile --windowed --name "PanoramaBridge" panoramabridge.py

The executable will be in the dist folder.

Alternative: Use Build Scripts

For easier building on Windows, use the provided build scripts:

Command Prompt: Run build_windows.bat
PowerShell: Run build_windows.ps1

These scripts automatically handle virtual environment setup, dependency installation, and executable creation. See BUILD_WINDOWS.md for complete details.

User Interface

Local Monitoring Tab

Directory Selection: Choose the local folder to monitor for new files
File Extensions: Specify which file types to monitor (e.g., raw, sld, csv, txt)
Subdirectory Monitoring: Option to include all subfolders
Directory Structure: Preserve local folder structure on remote server
File Stability: Configure how long to wait before considering a file complete

Advanced Settings Tab

File Monitoring Optimization:
- OS Events vs Polling: Uses efficient OS-level file system events by default
- Backup Polling: Optional backup polling for unreliable file systems (disabled by default)
- Polling Interval: Configurable 1-30 minute intervals when backup polling is enabled
Locked File Handling (Mass Spectrometer Workflows):
- Smart Detection: Automatically detects when files are locked by instruments during data acquisition
- Intelligent Retry: Configurable wait times and retry intervals for locked files
- Progress Indication: Shows elapsed wait time with countdown timers
- Initial Wait Time: Default 30 minutes before first retry (configurable)
- Retry Interval: Default 30 seconds between retry attempts (configurable)
- Max Retries: Default 20 attempts before giving up (configurable)
Performance Settings:
- Checksum Caching: Local caching for dramatic performance improvements (up to 1700x faster)
- Cache Management: Automatic cleanup and memory management
- Upload Verification: Configurable post-upload integrity checking

Remote Settings Tab

WebDAV Connection: Configure Panorama server connection
- URL: Your Panorama server (e.g., https://panoramaweb.org)
- Authentication: Username and password with secure storage option
- Connection testing with automatic endpoint detection
Remote Path Selection: Browse and select destination folders
Transfer Settings: Configure upload verification for integrity checking
Upload Verification: Enable/disable post-upload integrity checking

Transfer Status Tab

Queue Monitor: See how many files are waiting for transfer
Progress Tracking: Real-time progress bars for active uploads
Activity Log: Timestamped events and error messages
Log Access: View → View Application Logs for detailed troubleshooting

How PanoramaBridge Works: Step-by-Step Process

PanoramaBridge follows a systematic approach to monitor, verify, and transfer files to remote WebDAV servers. Here's the detailed process:

1. File Discovery and Monitoring

Directory Scanning Methods:

Primary Method: OS-level file system events using the watchdog library
- Monitors on_created, on_modified, and on_moved events
- Immediate detection with no polling overhead
- Handles file extensions: .raw, .wiff, .mzML, .mzXML, etc.
Backup Method: Optional polling (disabled by default)
- Configurable 1-30 minute intervals
- Uses os.walk() for recursive scanning or os.listdir() for single directory
- Only enabled when file system events are unreliable (network mounts, WSL2)

Extension Filtering:

Case-insensitive matching against user-configured extensions
Filters out hidden files (starting with . or ~)
Supports both recursive and non-recursive directory monitoring

2. File Stability Verification

Stability Detection:

Tracks file size changes over time using a pending files dictionary
Default 1-second stability timeout (file size unchanged)
Prevents uploading files still being written by instruments
Smart locked file handling for mass spectrometer workflows

Duplicate Prevention:

Maintains sets of queued_files and processing_files to prevent duplicates
Checks against remote paths to avoid re-uploading same destinations

3. Remote File Existence Check

WebDAV PROPFIND Method:

PROPFIND /remote/path/filename.raw HTTP/1.1
Host: panoramaweb.org
Depth: 0

Verification Process:

Sends HTTP PROPFIND request to check if file already exists on remote server
Retrieves remote file metadata: size, modification time, ETag
If file exists, compares with local file for conflict resolution
Creates necessary remote directories using WebDAV MKCOL method

4. Checksum Calculation

SHA256 Algorithm with Caching:

Method: SHA256 hash calculation using Python's hashlib library
Chunk Size: 256KB chunks for optimal memory/performance balance
Caching System: Local cache using file path + size + modification time as key
- Cache limit: 1,000 entries with automatic cleanup
- Cache invalidation on file size or modification time changes

Implementation:

hash_obj = hashlib.sha256()
with open(filepath, 'rb') as f:
    while chunk := f.read(256 * 1024):  # 256KB chunks
        hash_obj.update(chunk)
checksum = hash_obj.hexdigest()

5. File Upload Process

Adaptive Chunked Upload:

Chunk Size Determination (based on file size):
- Files < 100MB: 64KB chunks
- 100MB - 1GB: 256KB chunks
- 1GB - 5GB: 1MB chunks
- 5GB - 10GB: 2MB chunks
- Files > 10GB: 4MB chunks

Upload Methods:

Large Files (>100MB): Attempts Range request chunking
- Multiple HTTP PUT requests with Content-Range headers
- True progress tracking with real-time callbacks
Standard Files: Single HTTP PUT request with streaming
Progress Tracking: Real-time progress callbacks with bytes uploaded

HTTP Implementation:

PUT /webdav/remote/path/filename.raw HTTP/1.1
Host: panoramaweb.org
Content-Range: bytes 0-1048575/104857600
Content-Length: 1048576
Authorization: Basic <credentials>

[file chunk data]

6. Upload Verification System

Simple 3-Step Verification Process:

Step 1: File Existence & Size Check

Confirms remote file exists at expected path
Compares local and remote file sizes
Returns early if sizes don't match

Step 2: ETag Verification (Primary Method)

Attempts SHA256 ETag comparison first (most servers)
Falls back to MD5 ETag comparison (Apache default)
ETag mismatch indicates file difference and triggers conflict resolution

Step 3: Accessibility Check (Fallback)

Used only when ETag is unavailable or unknown format
Downloads first 8KB to verify file can be read
Confirms file exists, is readable, and user has permissions
Note: This is limited verification - cannot confirm complete file integrity

Implementation:

def verify_remote_file_integrity(self, local_filepath: str, remote_path: str, expected_checksum: str) -> tuple[bool, str]:
    """Verify remote file exists and matches expected local file using 3-step verification"""
    
    # Step 1: File existence and size comparison
    if local_size != remote_size:
        return False, f"size mismatch (local: {local_size}, remote: {remote_size})"
    
    # Step 2: ETag verification (primary method)
    if remote_etag and expected_checksum:
        clean_etag = remote_etag.strip('"').replace("W/", "")
        
        # Try SHA256 match first
        if clean_etag.lower() == expected_checksum.lower():
            return True, "ETag (SHA256 format)"
        
        # Try MD5 match for Apache servers
        elif len(clean_etag) == 32:
            local_md5 = hashlib.md5(open(local_filepath, 'rb').read()).hexdigest()
            if clean_etag.lower() == local_md5.lower():
                return True, "ETag (MD5 format)"
            else:
                return False, "ETag mismatch - file difference detected"
        
        # ETag mismatch with same length = file difference requiring resolution
        elif len(clean_etag) == len(expected_checksum):
            return False, "ETag mismatch - file difference detected"
    
    # Step 3: Accessibility check (fallback when ETag unavailable/unknown)
    head_data = self.webdav_client.download_file_head(remote_path, 8192)
    if head_data is None:
        return False, "cannot read remote file"
    
    if remote_etag is None:
        return True, "Size + accessibility (ETag unavailable)"
    else:
        return True, "Size + accessibility (unknown ETag format)"

7. Upload History and Remote Integrity Verification

Persistent Upload Tracking:

Maintains a JSON file (~/.panorama_upload_history.json) tracking all successful uploads
Records file path, size, checksum, and timestamp for each uploaded file
Enables intelligent skip-already-uploaded file detection across application restarts

On-Demand Remote Integrity Verification: PanoramaBridge uses a comprehensive 3-step verification system for remote integrity checks:

Startup Verification: Automatically checks all local files when monitoring starts to ensure they exist on remote server
Manual Verification: "Remote Integrity Check" button for on-demand verification of all files in the transfer table
Intelligent Conflict Resolution: Handles file differences through configurable conflict resolution settings
Automatic Recovery: Missing files are automatically queued for re-upload

Verification Results:

Verified: File exists and integrity confirmed
Missing: File not found on remote - automatically queued for re-upload
Changed/Conflict: File differs between local and remote - uses conflict resolution settings to determine action
Errors: Network/verification errors - logged for troubleshooting

Conflict Resolution Approach: When files differ between local and remote, PanoramaBridge no longer assumes corruption. Instead, it treats all differences as potential conflicts and applies your configured conflict resolution settings:

"Ask me each time": Shows dialog for user to choose action (Upload, Skip, etc.)
"Always upload": Automatically uploads the local version
"Always skip": Keeps the remote version unchanged
Other settings: Applied according to your configuration

This approach ensures that legitimate changes (whether local or server-side) are handled appropriately without assuming data corruption.

8. Metadata Storage and Integrity

Checksum Storage:

Stores checksums on WebDAV server as metadata files
File naming: filename.raw.checksum containing SHA256 hash
Used for future conflict resolution and integrity verification

Conflict Resolution:

Compares local checksum with stored remote checksum
User notification for checksum mismatches
Options to overwrite, skip, or manual resolution

8. Error Handling and Retry Logic

Locked File Handling:

Detection: Identifies files locked by mass spectrometers during acquisition
Smart Retry: Configurable wait times and retry intervals
- Default: 30-minute initial wait, 30-second retry interval, 20 max attempts
Progress Indication: Shows elapsed time and countdown timers
Status Messages: "File locked - waiting for instrument (5/30 minutes elapsed)"

Network Resilience:

Automatic retry for network failures
Persistent HTTP sessions for connection reuse
Timeout handling and connection management
Comprehensive error logging with detailed diagnostic information

This systematic approach ensures reliable, efficient, and verified file transfer while providing comprehensive monitoring and error handling for laboratory mass spectrometry workflows.

Application Features

File Monitoring

Real-time Detection: Monitors for new files as they're created
File Stability Check: Waits until files are fully written before transfer
Extension Filtering: Only processes files with specified extensions
Subdirectory Support: Optional recursive monitoring

WebDAV Transfer

Chunked Upload: Handles large files efficiently with configurable chunk sizes
Progress Tracking: Real-time progress bars for each transfer
Checksum Generation: Calculates SHA256 checksums for upload metadata (not used for verification due to performance cost)
Multi-Format ETag Verification: Supports SHA256 and MD5 ETags for efficient integrity checking
Accessibility Verification: Downloads first 8KB to verify file readability when ETags unavailable
Directory Structure: Option to preserve local folder structure on remote
Automatic Retry: Robust error handling and connection management

Security & Credentials

Secure Credential Storage: Uses system keyring for safe password storage
Support for Basic and Digest authentication
No plaintext password storage
Cross-platform keyring support with fallback options

Remote Directory Management

Remote Browser: Navigate WebDAV directories with GUI
Folder Creation: Create new folders on the remote server
Visual Directory Tree: Easy navigation and selection
Path Selection: Select destination paths intuitively

Menu System

View Menu:
- View Application Logs: Access detailed logs for troubleshooting
Help Menu:
- About: Application information and log file location

Configuration

Settings are automatically saved in:

Windows: %USERPROFILE%\.panoramabridge\config.json
Linux/Mac: ~/.panoramabridge/config.json

Credentials are stored securely in the system keyring.

Application logs are saved to: panoramabridge.log

Troubleshooting

Common Issues

Connection Problems

"Connection Failed" Error
- Verify Panorama server URL (usually https://panoramaweb.org)
- The application automatically tries /webdav endpoint if needed
- Check username and password
- Try both Basic and Digest authentication types
- Ensure server is accessible from your network
Keyring/Credential Storage Issues
- If you see "keyring backend not available" errors:
  - The application automatically installs keyrings.alt package
  - Restart the application after installation
  - Credentials will be saved securely on subsequent attempts

File Monitoring Issues

Files Not Being Detected
- Verify directory path is correct and accessible
- Check file extensions match exactly (case-insensitive)
- Ensure files are being created in the monitored directory
- Check subdirectory monitoring setting if files are in subfolders
- File stability timeout may need adjustment for large files
- WSL2 Users: For better file system event detection on Windows, use the native Windows build with .venv-win
Locked File Handling (Mass Spectrometer Workflows)
- "File locked - waiting for instrument": This is normal behavior when instruments are writing data
- Progress indication: Status shows elapsed wait time like "File locked - waiting for instrument (5/30 minutes elapsed)"
- Configuration: Adjust wait times in Advanced Settings → Locked File Handling
- Immediate retry: If you know a file is ready, restart monitoring to retry immediately
- Max retries exceeded: Increase max retries or check if instrument finished writing
- Troubleshooting: Check logs for "File locked during checksum" messages

Upload Problems

Upload Verification Issues
- "Verification failed" after successful upload:
  - Check network stability during verification
  - Verification uses ETag comparison when available, size + accessibility check as fallback
  - Disable verification in Transfer Settings if experiencing frequent issues
  - Manual verification: Use "Remote Integrity Check" button for on-demand verification
- Slow upload verification:
  - ETag verification is very fast (no download required)
  - Accessibility check downloads only 8KB for file readability test
  - Disable verification for maximum upload speed (less secure)
File Conflict Resolution
- "File conflict detected" messages: This is normal when files differ between local and remote
- Not corruption: The system no longer assumes files are corrupted when they differ
- Conflict handling: Uses your configured conflict resolution setting ("Ask me each time", "Always upload", etc.)
- User choice respected: When set to "Ask me each time", you'll see a dialog to choose the action
- Clear guidance: Messages explain what differences were detected and what actions will be taken
Folder Creation Failures (HTTP 403)
- Permission Denied: Most common issue
- Contact your Panorama administrator to request write permissions
- Try creating folders in directories where you have write access
- Check if you're in the correct user directory path
Path Encoding Issues (HTTP 409 - Path Conflict)
- Special Characters in Directory Names: Fixed in current version
- Previously, directories with @ symbols (like @files) could cause encoding issues
- The application now properly handles URL encoding/decoding for special characters
- If you encounter path conflicts, try refreshing the directory listing
Upload Failures
- Check available space on Panorama server
- Verify write permissions for the selected remote path
- Try reducing chunk size for problematic connections
- Check network connectivity and stability
- Monitor the application logs via View → View Application Logs

Performance Issues

Large File Problems
- Chunk sizes are automatically optimized based on file size
- Ensure stable network connection
- Check server timeout settings
- Monitor system memory usage during transfers

Panorama-Specific Notes

Server Configuration

Panorama WebDAV Endpoint: Typically /webdav (auto-detected)
Authentication: Usually Basic authentication
File Types: Commonly used for .raw, .mzML, .mzXML, .wiff files
Directory Structure: Often organized by project/experiment

Permissions

Write access must be granted by Panorama administrators
Users typically have access to their own directories
Project-specific permissions may apply
Contact your Panorama administrator for access issues

Logging and Diagnostics

Accessing Logs

In-Application: View → View Application Logs
Log File: panoramabridge.log in the application directory
Activity Tab: Real-time events in the Transfer Status tab

What to Include in Support Requests

Complete error messages from the application logs
Panorama server URL and username (no passwords)
File types and sizes being transferred
Network configuration details
Steps to reproduce the issue

Performance Optimization

For Large Files (>1GB)

Chunk sizes are automatically optimized (up to 4MB for files >10GB)
Use wired network connection
Transfer during off-peak hours
Monitor system resources

For Many Small Files

Automatic 64KB chunks optimize small file transfers
Monitor queue size in Transfer Status tab
Consider organizing files into batches

Network Optimization

Use stable, high-bandwidth connection
Check for network restrictions or firewalls
Monitor server load and availability
Consider VPN if accessing from outside institution

Additional Documentation

Technical Documentation

Checksum Caching Implementation - Details about the local checksum caching system that provides dramatic performance improvements
File Monitoring Optimization - Technical details about the optimized file monitoring system and performance benchmarks
File Monitoring Robustness - Thread safety and robustness improvements for file monitoring
Queue and Cache Implementation - Transfer queue management and persistent caching features

Development and Testing

Test Suite Documentation - Comprehensive test coverage and testing methodology
Test Setup Guide - Instructions for setting up and running tests

Build and Deployment

Windows Build Instructions - Complete guide for building Windows executables
GitHub Actions CI/CD - Automated builds and releases
Build Scripts Overview - Build automation and deployment tools

Demo Scripts and Examples

Demo Scripts Overview - Example scripts and diagnostic tools for development and testing

Support and Resources

Getting Help

Application Logs: First check View → View Application Logs for detailed error information
Test Connection: Verify settings with the "Test Connection" button
Panorama Documentation: Refer to your institution's Panorama setup guide
Administrator Contact: Reach out to your Panorama administrator for permissions

Common File Types for Panorama

Mass Spectrometry: .raw, .wiff, .mzML, .mzXML
Xcalibur Sequences: .sld (Sequence documents)
Proteomics: .fasta, .csv, .tsv, .txt
Analysis Results: .pdf, .xlsx, .zip

Name		Name	Last commit message	Last commit date
Latest commit History 72 Commits
.claude		.claude
.github/workflows		.github/workflows
.vscode		.vscode
build_scripts		build_scripts
demo_scripts		demo_scripts
docs		docs
panoramabridge.egg-info		panoramabridge.egg-info
screenshots		screenshots
tests		tests
.gitignore		.gitignore
.markdownlint.json		.markdownlint.json
.python-version		.python-version
AGENTS.md		AGENTS.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
markdown_lint.py		markdown_lint.py
panoramabridge.py		panoramabridge.py
pyproject.toml		pyproject.toml
pytest.ini		pytest.ini
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
run_tests.sh		run_tests.sh

Folders and files

Latest commit

History

Repository files navigation

PanoramaBridge

Quick Links

Features

Core Features

Advanced Features

Performance & Reliability

Requirements

Python Dependencies

Installation

Prerequisites: Install Python

Option 1: Install from PyPI (Recommended)

Option 2: Install from Source

Option 3: Download Windows Executable

Quick Start

Creating a Windows Executable (Optional)

1. Install PyInstaller

2. Create the Executable

Alternative: Use Build Scripts

User Interface

Local Monitoring Tab

Advanced Settings Tab

Remote Settings Tab

Transfer Status Tab

How PanoramaBridge Works: Step-by-Step Process

1. File Discovery and Monitoring

2. File Stability Verification

3. Remote File Existence Check

4. Checksum Calculation

5. File Upload Process

6. Upload Verification System

7. Upload History and Remote Integrity Verification

8. Metadata Storage and Integrity

8. Error Handling and Retry Logic

Application Features

File Monitoring

WebDAV Transfer

Security & Credentials

Remote Directory Management

Menu System

Configuration

Troubleshooting

Common Issues

Connection Problems

File Monitoring Issues

Upload Problems

Performance Issues

Panorama-Specific Notes

Server Configuration

Permissions

Logging and Diagnostics

Accessing Logs

What to Include in Support Requests

Performance Optimization

For Large Files (>1GB)

For Many Small Files

Network Optimization

Additional Documentation

Technical Documentation

Development and Testing

Build and Deployment

Demo Scripts and Examples

Support and Resources

Getting Help

Common File Types for Panorama

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 10

Packages 0

Uh oh!

Contributors

Uh oh!

Packages