Add MIT License and unified CLI#13
Merged
Merged
Conversation
Major Features:
- Centralized filename patterns (FILENAME_PATTERNS dictionary)
- Improved CV summary with informative support statistics
- Complete ROC/PR curve generation for individual folds and CV summary
- Enhanced confusion matrix CSV with row labels
- Metrics visualization dashboard (4-panel summary)
Filename Changes:
- Renamed from {metric}_fold_{fold} to fold_{fold:02d}_{metric}
- Added mean/std/n_folds to CV summary filenames
- Consistent naming across all outputs
Data Quality:
- Confusion matrix CSVs now include row labels (index=True)
- CV summary support shows: mean ± std (total=X)
- Classification report with mean ± std for all metrics
Bug Fixes:
- Fixed string label handling in ROC/PR curves
- Fixed path resolution to prevent _out directories
- Fixed DataFrame → dict conversion for classification reports
- Fixed imports (sns graceful fallback, scipy.interp removal)
- Fixed 1D probability array handling
Files Modified:
- _SingleClassificationReporter.py: Core reporter logic
- _Plotter.py: Visualization functions
- storage.py: File saving with proper paths
- plot_*.py: ROC/PR/confusion matrix plotting
- _calc_conf_mat.py: Confusion matrix calculation
🤖 Generated with Claude Code (https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changed from pure snake_case to structured naming with: - Hyphens within logical chunks (fold-00, confusion-matrix) - Underscores between major components (fold-00_confusion-matrix) Benefits: - Clear visual hierarchy - Better readability within compound terms - Logical grouping of related information Examples: - fold-00_balanced_accuracy-0.500.json - fold-00_confusion-matrix_bacc-0.500.csv - folds-all_roc-curve_auc-0.700_0.305_n-5.jpg 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Created scitex.ai.feature_selection module for ML feature analysis - extract_feature_importance: Auto-detect tree-based/linear models - aggregate_feature_importances: Cross-fold statistics (mean, std, CV) - select_features_univariate: ANOVA F-test feature selection - create_feature_importance_dataframe: Structured output - Enhanced ClassificationReporter with automatic feature importance - Accepts model and feature_names in calculate_metrics() - Auto-extracts importance when model provided - Aggregates across CV folds with stability metrics - Saves per-fold JSON files - Integrated feature importance into all report formats - Added section between Summary Performance and Visualizations - Shows mean, std, min, max, CV for each feature - Includes explanatory note about coefficient of variation - Works with org, markdown, HTML, LaTeX, DOCX - Added feature importance plotting capabilities - Bar plots with error bars for cross-fold analysis - Heatmaps for importance across folds - Stability visualization with CV metrics - Refactored plot module naming convention - Renamed plot_*.py to _plot_*.py for consistency - Updated imports and __init__.py exports 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…rganization - Add automatic project creation (no need for --create-project) - Create project symlinks during enrichment with CC-IF-Year-Author-Journal naming - Organize BibTeX files in info/bibtex/ directory with automatic merged.bib - Move project_metadata.json to info/ directory - Add bibliography.bib symlink at project root for easy access - Fix asyncio issues with nest-asyncio - Improve initialization message to show library path - Support format inference for --export (e.g., --export file.bib) Co-Authored-By: Claude <noreply@anthropic.com>
- Implement ParallelPDFDownloader with multiple Chrome workers - Add publisher-aware scheduling to avoid rate limits - Automatically disable parallel when no authentication available - Configure max_parallel workers via config cascade system - Intelligently group papers by publisher/journal for optimal scheduling - Adjust worker count based on publisher diversity in batch - Add publisher identification from DOI prefixes and URLs Configuration: - Set SCITEX_SCHOLAR_PDF_MAX_PARALLEL env var (default: 3) - Set SCITEX_SCHOLAR_PDF_USE_PARALLEL env var (default: true) - Or configure in config.yaml under pdf_download section Co-Authored-By: Claude <noreply@anthropic.com>
- Update BrowserManager import to ScholarBrowserManager - Fix config.get() calls to handle ScholarConfig properly - Add browser_mode parameter to ScholarBrowserManager initialization - Sequential downloads work correctly with open-access papers
Experimental work to solve "NO PDF URLS FOUND" issue for paywalled papers with institutional access. **Proof of Concept**: - ✅ Popup window capture works (tested on IEEE paper) - ✅ Can click OpenURL JavaScript links - ✅ Can access publisher pages after authentication - ✅ Can identify PDF elements on publisher pages **New Files**: - url/TODO.md: Implementation plan for OpenURL strategy - url/_test_popup_capture.py: Working popup capture test - url/_test_full_pdf_download.py: Full workflow test - .dev/access_strategy_experiments/: All experimental code & findings **Key Finding**: OpenURL resolver uses JavaScript links that open popups. We need to: 1. Click the link (e.g., "IEEE Electronic Library") 2. Capture the popup window 3. Apply publisher-specific strategy (extract article#, build PDF URL) **Next Steps** (see url/TODO.md): - Integrate popup handling into ScholarURLFinder - Create publisher strategy pattern (IEEE, Elsevier, IOP) - Test end-to-end on failing papers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…oads Implemented a complete visual debugging system with stacking popup messages and automatic screenshot capture across all browser automation components. **Key Features**: - Stacking popup messages that persist across page navigations - Automatic timestamped screenshots at every step (millisecond precision) - Network idle strategy instead of fixed delays for adaptive timing - Comprehensive visual coverage across Auth Gateway, OpenURL, URL Finder, and Download methods **Authentication Gateway Pattern**: - New AuthenticationGateway class for transparent authentication preparation - Config-based publisher detection using DOI prefixes (IEEE, Springer, Nature, etc.) - Visits OpenURL to establish publisher-specific session cookies before download - Caches authentication state to avoid redundant operations **Visual Tracking Components**: 1. AuthenticationGateway: Session establishment messages 2. OpenURLResolver: Publisher link finding and navigation (10+ messages) 3. URL Finder: PDF URL discovery strategies (8 messages per attempt) 4. Download Methods: All 3 methods with detailed progress (20+ messages total) - Direct Download (5 messages) - Chrome PDF Viewer (12+ messages with extra patience) - Response Body (8 messages) **Chrome PDF Viewer Improvements**: - Removed redundant networkidle wait that caused page timeouts - Added IEEE-specific detection methods for PDF viewer elements - Increased patience: 10s extra wait after networkidle for PDF viewer initialization - Network idle waits instead of fixed delays (up to 30s, adaptive) - Enhanced detection with 6 different PDF viewer detection methods **Screenshot System**: - Automatic capture at every popup message - Filenames: `YYYYMMDD_HHMMSS_mmm_<message_text>.png` - Organized in `~/.scitex/scholar/workspace/screenshots/visual_tracking/` - Creates complete visual timeline for post-mortem analysis - Non-blocking: failures don't break downloads **Popup Message System**: - Messages stack vertically (up to 10 visible) - Each message persists for 60 seconds - Automatic re-injection after page navigations via framenavigated handler - Consistent prefixes: Auth Gateway, OpenURL, URL Finder, Chrome PDF, etc. - Status indicators: ✓ (success), ✗ (failure) **Configuration**: - Added authentication section to default.yaml with paywalled publishers - DOI prefix patterns for automatic publisher detection - Domain patterns for URL-based authentication requirements **Testing**: - Successfully tested with IEEE paywalled papers (2.08 MB download verified) - Visual timeline shows complete flow from authentication to download - Screenshots provide clear evidence of each step **Cleanup**: - Removed old url_v01 directory (deprecated implementation) - Removed obsolete Zotero translators directory (700+ files) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…with categorization
Moved universal Playwright utilities from Scholar-specific location to SciTeX-wide
scope, organized by category for better maintainability and reusability.
## New Structure
src/scitex/browser/
├── debugging/ Visual debugging utilities
│ ├── show_popup_and_capture.py (special versatile function)
│ ├── show_grid.py
│ └── highlight_element.py
├── pdf/ Chrome PDF viewer utilities
│ ├── detect_chrome_pdf_viewer.py
│ └── click_download_for_chrome_pdf_viewer.py
└── interaction/ Click and fill helpers
├── click_center.py
├── click_and_wait.py
├── click_with_fallbacks.py
└── fill_with_fallbacks.py
## Key Changes
- Elevated 9 universal utilities from scholar.browser.utils to scitex.browser
- Organized utilities into debugging/, pdf/, and interaction/ categories
- Removed _async suffix from function names, added backward compatibility aliases
- Updated Scholar imports to re-export from elevated location
- Improved docstrings and variable names for clarity
- Scholar-specific utilities (take_screenshot, wait_redirects, close_unwanted_pages,
JSLoader) remain in scholar.browser.utils
## Benefits
- Universal utilities available across entire SciTeX ecosystem
- Better organization and discoverability by category
- Maintained full backward compatibility
- show_popup_and_capture properly recognized as ecosystem-wide tool
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Document new scitex.browser module in main README - Add browser utilities section to What's New - Update submodules table with browser entry Documentation highlights: - Debugging utilities (show_popup_and_capture, show_grid, highlight_element) - PDF utilities (detect_chrome_pdf_viewer, click_download_for_chrome_pdf_viewer) - Interaction utilities (click_and_wait, click_with_fallbacks, fill_with_fallbacks) - Usage examples for common automation patterns 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
**Key Changes:** - Renamed all async browser utilities to use _async suffix consistently - show_popup_and_capture -> show_popup_and_capture_async - highlight_element -> highlight_element_async - detect_chrome_pdf_viewer -> detect_chrome_pdf_viewer_async - click_download_for_chrome_pdf_viewer -> click_download_for_chrome_pdf_viewer_async - click_with_fallbacks -> click_with_fallbacks_async - fill_with_fallbacks -> fill_with_fallbacks_async - click_center -> click_center_async **Backward Compatibility:** - Scholar module provides aliases for old names without _async suffix - No breaking changes for existing Scholar code **Documentation:** - Updated all READMEs with new function names - Updated all code examples - Clarified naming convention **Architecture Decision:** - Moved click_and_wait back to Scholar (uses Scholar-specific wait_redirects) - Removed click_and_wait from browser.interaction (not universal) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
**Cleanup:** - Remove duplicate universal utilities from scholar/browser/utils - Keep only Scholar-specific utilities (click_and_wait, wait_redirects, etc.) - Add comprehensive README documenting Scholar-specific vs universal utilities **Config Integration:** - Load auth endpoint patterns from config (authentication.auth_endpoint_patterns) - Load article URL patterns from config (authentication.article_url_patterns) - Fallback patterns if config fails to load - Cached config loading for performance **Benefits:** - Single source of truth for auth patterns (config YAML) - Easy to extend auth patterns without code changes - Clear separation: Scholar-specific vs universal utilities 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Renamed handle_popups_async → close_popups_async for clarity. Changes: - Moved Scholar's _handle_popups_async.py to scitex.browser.interaction/close_popups.py - Updated all imports to use close_popups_async - Updated documentation in all READMEs - Removed incorrect reference to _close_popups_async.py in Scholar README 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Fixed NameError issues caused by incorrect backward compatibility aliases. Changes: - Removed aliases that referenced non-existent function names - Renamed click_center → click_center_async for consistency - All async functions now use _async suffix consistently Files fixed: - src/scitex/browser/debugging/_show_grid.py - src/scitex/browser/pdf/detect_chrome_pdf_viewer.py - src/scitex/browser/pdf/click_download_for_chrome_pdf_viewer.py - src/scitex/browser/interaction/click_center.py Verified all imports work correctly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…apture_async Marked Scholar's take_screenshot as deprecated with migration guidance. Rationale: - show_popup_and_capture_async provides BOTH visual feedback AND screenshots - Universal function works across all browser automation workflows - Better debugging experience with stacking popup messages Migration path documented in: - _take_screenshot.py docstring - Scholar browser utils README Existing code continues to work, but new code should use show_popup_and_capture_async. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
ScholarOrchestrator now accepts paper titles in addition to DOIs.
Uses ScholarEngine to resolve title → DOI → full metadata.
Example:
python -m scitex.scholar.core.ScholarOrchestrator \
--doi-or-title "Seizure Forecasting by High-Frequency Activity" \
--project pac \
--chrome-profile system
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Provides async-compatible disk caching using joblib.Memory. Complements existing cache_disk for synchronous functions. Note: Has event loop limitations with nested async calls. For now, ScholarEngine uses its own JSON-based caching which works reliably. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Show temp.pdf → main.pdf rename step in logs for clarity. Makes it clear that final PDF is stored as main.pdf. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
All getters use PATH_STRUCTURE.format() consistently: - get_library_project_entry_dirname() - formats symlink names - get_library_project_entry_pdf_fname() - formats PDF filenames - get_library_project_entry_dir() - entry directory path - get_library_project_entry_metadata_json() - entry metadata path - get_library_project_entry_logs_dir() - entry logs path Makes PATH_STRUCTURE the single source of truth for all naming. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Moved entry name formatting from ScholarOrchestrator to PaperIO. PaperIO.get_entry_name_for_project() uses PathManager getters. ScholarOrchestrator._link_to_project() now 24 lines (was 66). Single source of truth: PATH_STRUCTURE formats all names. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implemented parallel paper processing system with the following features:
Core Components:
- ScholarPipelineParallel: Orchestrates parallel downloads with N workers (default: 4)
- ScholarPipelineBibTeX: Processes BibTeX files through parallel pipeline
- ScholarPipelineSingle: Single paper pipeline with per-paper logging
Key Features:
- Worker count auto-capped by number of papers (2 papers = 2 workers, not 4)
- Pre-authentication verification before spawning workers
- Chrome profile syncing for each worker (system_worker_0, system_worker_1, etc.)
- Per-paper logging to MASTER/{paper_id}/logs/pipeline.log
- Resumable processing (skips already downloaded papers)
- Title-based search support (resolves DOI from title)
Logging Improvements:
- Added logger.to(file_path) context manager for clean temporary logging
- Added log_to_file() context manager in scitex.logging._context
- Moved Tee from scitex.gen to scitex.logging with lazy logger to avoid circular imports
- Fixed circular import in _latex_fallback.py using lazy logger pattern
Bug Fixes:
- ChromeProfileManager: Handle comma-formatted numbers in rsync output (e.g., "3,301")
- ScholarPipelineSingle: Convert citation year fields "2015" → "y2015" for Pydantic model
- PaperIO: Normalize journal names in PDF filenames ("Sci Rep" → "SciRep")
Test Results:
- Successfully processed 23/30 papers (77% success rate) with 8 workers
- Clean error handling with per-paper logs for debugging failures
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Moved Tee class to scitex.logging for better organization. Changes: - Moved scitex.gen._tee → scitex.logging._Tee - Updated gen/__init__.py to remove Tee exports - ScholarEngine: Use stx.io.load/save for cache instead of json This consolidates logging utilities in one place. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Minor updates to session lifecycle management. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Remove ScholarOrchestrator.py (replaced by pipeline system) - Remove old CLI documentation - Update TODO.md with parallel pipeline status - Update default.yaml configuration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…g files - Enhanced config loader to support _categories subdirectory - Added categorized YAML config files for better organization - Updated TODO.md with removed completed tasks - Added migration note to config README Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Updated TODO.md with clarifications on config system and documentation - Removed TODO_v01.md and TODO_v02.md as no longer needed Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements scitex CLI with local-first, cloud-optional architecture: - Fix entry point: scitex.cli:main → scitex.cli:cli - Add scitex cloud commands (wraps tea for Gitea operations) - list, clone, create, delete, search, fork - Auto-detection of repository owners - Consistent --login options - Add scitex scholar commands (wraps existing scholar module) - single, parallel, bibtex processing - Browser automation with institutional access - Local library management - Enable -h/--help flags across all commands Philosophy: Local-first execution, cloud provides optional sync/convenience. Users control their resources. No server dependency for core features. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add MIT License file with proper copyright attribution - Add click dependency to requirements.txt for CLI functionality 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
ywatanabe1989
added a commit
that referenced
this pull request
Jan 5, 2026
Add MIT License and unified CLI
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
🤖 Generated with Claude Code