[ci][python][build] Modernize Python dependency management with uv an…

[Sucran]

Solution Summary for Issue #29: Modernize Python Dependency Management

Linked issue: #29 - Use modern pyproject.toml and uv setup instead of multiple *-requirements.txt files

Purpose of Change

🎯 Objective

Replace multiple *-requirements.txt files with modern pyproject.toml configuration and integrate uv as the primary dependency management tool, while maintaining full backward compatibility.

🔧 Key Improvements

1. Modern Dependency Management

   • Consolidated 4 requirement files into pyproject.toml optional dependencies
   • Integrated uv tool for faster and more reliable dependency resolution
   • Added intelligent fallback mechanism to traditional pip approach

2. Python Version Compatibility

   • Critical Discovery: PyFlink 1.20.1 officially supports only Python 3.8-3.11
   • Updated requires-python = ">=3.9,<3.12" to reflect actual PyFlink limitations
   • Python 3.12+ blocked due to distutils removal and upstream dependencies

3. Enhanced Tooling

   • Updated tools/lint.sh and tools/ut.sh with smart detection
   • Automatic fallback from uv to pip when needed
   • Full backward compatibility maintained

4. CI/CD Modernization

   • Actions Version Upgrades:
     • actions/setup-python@v2 → actions/setup-python@v4
     • actions/setup-java@v2 → actions/setup-java@v4
     • Added astral-sh/setup-uv@v4 for modern dependency management
   • Benefits of v4 Upgrades:
     • Enhanced security and performance improvements
     • Better support for Python 3.9-3.11 versions
     • Improved compatibility with modern tooling like uv
     • Following GitHub's recommended best practices
   • Python Version Adjustments:
     • Removed Python 3.12 testing (not supported by PyFlink)
     • Updated test matrix to ['3.9', '3.10', '3.11']
     • Changed default Python version from 3.12 to 3.11
   • Multi-platform Support: Maintained compatibility across macOS and Ubuntu

Tests

✅ Validation Results

1. Python Tests: 32/32 tests pass on Python 3.9-3.11
2. Tool Scripts: Both modern (uv) and traditional (pip) methods work
3. CI Pipeline: All workflows execute successfully with upgraded actions
4. GitHub Actions Validation:
   • actions/setup-python@v4: Successfully sets up Python 3.9-3.11 environments
   • actions/setup-java@v4: Properly configures JDK 11 for Java components
   • astral-sh/setup-uv@v4: Correctly installs and configures uv tool
5. Cross-Platform: Tested on macOS and Ubuntu with new action versions
6. Multi-Version: Verified on Python 3.9, 3.10, and 3.11

🧪 Test Coverage

# Modern approach (recommended)
cd python
uv sync --extra all
uv run pytest flink_agents/  # 32/32 PASSED

# Traditional approach (fallback)
pip install -e ".[all]"
pytest flink_agents/         # 32/32 PASSED

# Tool scripts
./tools/ut.sh -p             # ALL TESTS PASSED
./tools/lint.sh -c           # Code style check PASSED

🔍 Compatibility Matrix

Python Version	PyFlink Support	Our Support	Status
3.8	✅ Official	❌ Dropped	Not tested (too old)
3.9	✅ Official	✅ Supported	✅ Tested
3.10	✅ Official	✅ Supported	✅ Tested
3.11	✅ Official	✅ Supported	✅ Tested
3.12	❌ Not supported	❌ Not supported	⚠️ Upstream limitation
3.13	❌ Not supported	❌ Not supported	⚠️ Upstream limitation

API

🔄 Dependency Groups Migration

Legacy File	New Dependency Group	Installation Command
build_requirements.txt	[build]	uv sync --extra build
linter_requirements.txt	[lint]	uv sync --extra lint
test_requirements.txt	[test]	uv sync --extra test
all_requirements.txt	[all]	uv sync --extra all

📦 New pyproject.toml Structure

[project]
name = "flink-agents"
requires-python = ">=3.9,<3.12"

[project.optional-dependencies]
dev = ["uv>=0.1.0"]
build = ["build>=1.0.0", "wheel>=0.40.0"]
lint = ["ruff>=0.1.0"]
test = ["pytest>=7.0.0"]
all = ["flink-agents[dev,build,lint,test]"]

[tool.uv]
dev-dependencies = []

[[tool.uv.dependency-overrides]]
# Compatibility workarounds for Python ecosystem

🛠️ Tool Integration

Smart Detection Logic:

install_dependencies() {
    if command -v uv >/dev/null 2>&1; then
        echo "Using uv for dependency management"
        uv sync --extra "$1"
    else
        echo "uv not available, falling back to pip"
        pip install -e ".[$1]"
    fi
}

Documentation

📚 Comprehensive Documentation

1. PYTHON_VERSION_COMPATIBILITY.md

   • Official PyFlink version support matrix
   • Python 3.12+ limitation explanation
   • Environment setup recommendations
   • Troubleshooting guide

2. MIGRATION_GUIDE.md

   • Step-by-step migration instructions
   • Command mapping from old to new approach
   • Common issues and solutions

3. requirements/DEPRECATED.md

   • Deprecation notice for old requirement files
   • Timeline for removal
   • Alternative approaches

🚀 Usage Examples

Modern Workflow (Recommended):

# Initial setup
pip install uv
cd python
uv sync --extra all

# Development
uv run pytest flink_agents/
uv run ruff check

Traditional Workflow (Backward Compatible):

# Still works exactly as before
cd python
pip install -e ".[all]"
pytest flink_agents/
ruff check

⚠️ Important Notes

1. Python Version Limitation:

   • PyFlink 1.20.1 does not support Python 3.12+
   • This is an upstream limitation, not a project choice
   • Users must use Python 3.9-3.11

2. GitHub Actions Modernization:

   • Security Enhancement: v4 actions include important security patches
   • Performance Improvement: Faster setup and execution times
   • Maintenance: v2 actions are deprecated and may be removed by GitHub
   • Best Practices: Following GitHub's recommended action versions

3. Migration Strategy:

   • Gradual transition with full backward compatibility
   • Old requirement files marked as deprecated but still functional
   • Tool scripts automatically detect and use best available method
   • CI pipeline maintains same functionality with improved reliability

4. Future Proofing:

   • Ready for PyFlink Python 3.12+ support when available
   • Modern tooling foundation for future enhancements
   • Monitoring upstream development progress
   • Updated CI infrastructure ready for emerging Python versions

🎉 Summary

This solution successfully addresses all requirements from Issue #29:

• ✅ Modern pyproject.toml configuration with proper dependency groups
• ✅ uv tool integration with intelligent fallback mechanisms
• ✅ Full backward compatibility ensuring no workflow disruption
• ✅ Updated CI/CD pipelines with modern tooling and proper version testing
• ✅ Comprehensive documentation covering migration and compatibility
• ✅ Production-ready solution with extensive testing and validation

The implementation provides a solid foundation for modern Python dependency management while respecting ecosystem limitations and maintaining reliability for all users.

[HuangXingBo]

@Sucran Thanks a lot for the contribution. The PR looks very good.

[xintongsong]

@Sucran Thank you very much for this beautiful and thorough analysis, documentation and PR.