feat: modernize project tooling and add comprehensive type annotations#282
Merged
feat: modernize project tooling and add comprehensive type annotations#282
Conversation
|
|
mlwelles
changed the title
Add Makefile with common development tasks: - setup: Install tools and sync dependencies - sync: Sync virtual environment dependencies - check: Run pre-commit hooks on all files - protogen: Regenerate protobuf files - test: Run pytest tests Add .python-version to set Python 3.13 as canonical dev version Add .pre-commit-config.yaml with comprehensive hooks for code quality Python version changes: - Removed support for EOL Python 3.7 and 3.8 - Added support for Python 3.13 and 3.14
Update Python version requirements: - Minimum runtime version: Python 3.9+ (was 3.7+) - Canonical development version: Python 3.13+ - Remove support for EOL Python 3.7 and 3.8 - Add support for Python 3.13 and 3.14 Update grpcio dependency to avoid yanked version: - Change from >=1.65.0 to >=1.65.1 Add comprehensive dev dependencies for type checking and code quality: - mypy, mypy-protobuf: Type checking and proto stub generation - grpc-stubs, types-grpcio, types-protobuf: gRPC type stubs - pre-commit: Git hook management - shellcheck-py: Shell script linting - types-requests, pydantic-settings: Additional type stubs Configure mypy with strict type checking and proto plugin support
Update protobuf generation script to require Python 3.13+: - Add version checks for Python 3.13+ and grpcio 1.66.2+ - Remove old Python 3.13 restriction (now required) - Provide helpful error messages with setup instructions Add mypy type stub generation: - Generate .pyi files with --mypy_out and --mypy_grpc_out flags - Enable better type checking for generated protobuf code - Add mypy-protobuf dependencies to pre-commit mypy hook The script now validates the development environment and provides clear instructions if dependencies need to be updated.
Update development setup documentation: - Document uv-based workflow for project setup - Note Python 3.13+ requirement for development - Update protobuf regeneration instructions - Change all python invocations to use 'uv run python' Update README.md: - Add instructions for make setup and make sync - Document Python version requirements (3.9+ runtime, 3.13+ dev) - Update protogen section with new commands Update example READMEs: - Change python commands to uv run python Update PUBLISHING.md: - Reference make protogen for regenerating protobufs
Fix shell script issues identified by shellcheck: - Quote array expansion to avoid re-splitting elements - Declare and assign SRCDIR separately to avoid masking return values - Quote command substitution to prevent word splitting
- Consolidate ruff configuration in pyproject.toml - Remove deprecated .trunk/configs/ruff.toml file - Configure mypy with stricter settings (currently commented for gradual adoption): * check_untyped_defs enabled * no_site_packages to avoid checking dependencies * Exclude generated proto files from strict checks - Update ty and mypy hooks with required dependencies (grpcio, protobuf, pytest) - Configure ty to check only pydgraph and tests directories - Ignore example notebooks in trunk linting
- Remove Python 2 compatibility code: * Remove urlparse fallback in client_stub.py * Simplify is_string() without basestring check * Remove basestring and long type checks from tests - Fix code quality issues: * Fix bare except clauses (use Exception instead) * Fix ambiguous variable name (l -> last) * Fix undefined variable references in txn.py * Add return type annotation to convert.py * Add future annotations import for X | Y union syntax - Export pydgraph.open function in __all__ - Remove wildcard imports, use explicit imports - Fix protobuf stub imports (absolute -> relative) - Update protogen.py to fix stub imports automatically - Exclude generated proto files and scripts from mypy - Configure ty to exclude generated proto files All changes maintain backwards compatibility for Python 3.9+
Remove scripts from mypy exclude list to ensure type annotations are required for script files when strict checking is enabled.
Add type annotations to all Python files in pydgraph package, tests, and scripts. Enable mypy strict mode with disallow_untyped_defs and disallow_incomplete_defs to ensure all functions have proper type signatures. This improves IDE support, catches type errors at development time, and makes the codebase more maintainable. Changes: - Add type annotations to pydgraph core modules (client, txn, client_stub) - Add type annotations to utility modules (util, errors, convert) - Add type annotations to all test files and helper modules - Add type annotations to scripts/protogen.py - Enable strict mypy checking in pyproject.toml - Use modern union syntax (X | None) via __future__ annotations - Configure mypy to skip type checking of generated proto modules - Add mypy overrides for third-party packages without stub support
Add deps-docker task to check and install Docker on Darwin and Linux platforms. Update test task to use existing local-test.sh script with Docker Compose for proper test isolation with dynamic port allocation. Changes: - Add deps-docker task that verifies Docker and Docker Compose v2 installation - Update deps task to include deps-docker - Update test task to depend on deps-docker and use local-test.sh - Fix test_connect.py to handle TEST_SERVER_ADDR with or without port - Use existing docker-compose.yml in tests/ directory for test infrastructure This ensures tests run in isolated Docker containers with no port conflicts, following the existing test infrastructure pattern.
Add deps-ty task to install ty type checker and create GitHub Actions workflow for mypy type checking on pull requests. Changes: - Add deps-ty task to check and install ty type checker - Update deps to depend on deps-ty - Add mypy.yml workflow for CI type checking - Workflow runs mypy with pyproject.toml configuration on all PRs This ensures type checking tools are available and type checks run automatically on all pull requests.
Add GitHub Actions workflow with two parallel jobs for code quality checks and running tests. Uses best practice actions for setup. Jobs: - Check Code Quality: runs make check with pre-commit hooks - Run Tests: runs make test with Docker-based test infrastructure Setup includes: - Python 3.13 via actions/setup-python - uv package manager via astral-sh/setup-uv with caching - ruff and ty type checkers installed automatically - Docker available by default on ubuntu-latest runners This provides comprehensive CI coverage for all PRs and commits to main.
Update test workflow to run tests across all supported Python versions using a matrix strategy. Each Python version (3.9, 3.10, 3.11, 3.12, 3.13, 3.14) gets its own test job with proper version pinning. Changes: - Use matrix strategy for python-version - Set up Python with correct version for each job - Pin Python version using uv python pin before running tests - Display Python version in job name for clarity This ensures compatibility across all supported Python versions.
Updated __maintainer__ fields from "Hypermode Inc." and "Animesh Pathak" to "Istari Digital <contact@istaridigital.com>" across all Python files. Also updated project Homepage URL from hypermodeinc to dgraph-io organization.
- Regenerate protobuf files with mypy type stubs (.pyi files) - Add TYPE_CHECKING imports to fix forward reference issues - Fix import ordering to comply with ruff E402 requirements - Update pyproject.toml: * Add ruff exclude for notebooks and embeddings examples * Remove mypy_protobuf.main plugin (stubs generated separately) * Update mypy exclude configuration - Move module docstrings before __future__ imports per PEP 8 - Ensure all type annotations pass strict mypy checks - All tests passing (91/91) Resolves type checking issues after merging main branch changes.
mlwelles
force-pushed
the
mlwelles/modernize-project-and-improve-typing
branch
from
ddd8cb5 to
15e8146
Compare
Ensures pytest is run from the virtual environment in CI environments where pytest may not be globally available.
The protobuf files were generated with protoc using protobuf 6.x, which is incompatible with protobuf 5.x runtime. Update the minimum version requirement to ensure compatibility.
grpcio-tools < 1.66.0 requires protobuf < 6.0, which conflicts with the protobuf 6.x generated files. Update to grpcio-tools >= 1.66.2 for all Python versions to resolve the dependency conflict.
Protogen requires Python 3.13+ due to grpcio-tools >= 1.66.2 requirement. Update CI workflow to only run protobuf verification on Python 3.13.
protoc-gen-mypy is not accessible in PATH when using pip install, causing verification to fail. The protobuf files are committed to the repository, so this check is not critical for CI.
Auto-detect uv availability and fall back to pytest if uv is not installed. This allows the script to work in both local development (with uv) and CI environments (without uv).
…oject-and-improve-typing # Conflicts: # tests/test_namespace.py
Changes dependency constraints to support protobuf 4.23.0 through 6.x, allowing users to choose the version that works best for their environment. Changes: - Update protobuf dependency: >=6.0.0 → >=4.23.0,<7.0.0 - Update grpcio dependency: >=1.65.1 → >=1.65.0 (minor adjustment) - Add protobuf compatibility documentation to README - Update Python version requirement in README: 3.7 → 3.9 This provides backwards compatibility while still supporting modern protobuf versions. Users can pin to specific protobuf versions if needed for compatibility with other packages in their environment.
Add noqa comments for intentional builtin shadowing in public API: - open() function (A004, A001) - established API for opening connections - ConnectionError class (A004, A001) - custom exception type - vars parameter (A002) - required for backward compatibility These shadow builtins but are part of the stable public API and cannot be renamed without breaking existing code.
Fix TRY300 and TRY201 linting issues: - Move return statements outside try blocks to avoid catching unintended exceptions from return value access - Use bare 'raise' instead of 'raise error' to preserve stack traces - Consolidate duplicate return logic into single statements This improves code clarity and ensures exceptions are only caught from the intended operations (gRPC calls), not from accessing response attributes.
Remove quick start instructions from README and just link to CONTRIBUTING.md for all contribution guidelines.
- Add blank lines around fenced code blocks (MD031) - Add language specifiers to code blocks (MD040)
Update SPDX copyright headers across all Python files to reflect 2026.
Contributor
Author
DGR-152: Removed Dgraph Cloud ReferencesChanges MadeDgraph Cloud no longer exists, so all cloud-specific functionality and documentation has been removed from pydgraph. Code ChangesRemoved cloud-specific methods:
Cleanup:
Documentation ChangesREADME.md:
Examples:
Deprecated Notebooks:
These two notebooks cannot be easily updated as they rely heavily on cloud-specific features (Lambda functions, cloud authentication, cerebro API) that no longer exist. Migration Path for UsersUsers who were using Before (deprecated): client_stub = pydgraph.DgraphClientStub.from_cloud(
"endpoint.grpc.cloud.dgraph.io:443",
"api-key"
)After (recommended): # Using connection string
client = pydgraph.open("dgraph://endpoint:9080")
# Or using standard stub with SSL
creds = grpc.ssl_channel_credentials()
client_stub = pydgraph.DgraphClientStub("endpoint:443", creds)
client = pydgraph.DgraphClient(client_stub)Testing
Commit: 721f2a5 |
Dgraph Cloud no longer exists, so this removes all cloud-specific functionality and documentation: - Removed from_cloud() static methods from DgraphClientStub and AsyncDgraphClientStub - Removed parse_host() helper methods and unused urlparse imports - Updated README.md to remove cloud connection section and examples - Removed apikey parameter from connection string documentation - Simplified examples/embeddings/computeEmbeddings.py - Updated RAG notebook to use standard connection - Added deprecation warnings to cloud-dependent notebooks (dgraph-episode1.ipynb and dgraph-ai-classification.ipynb) - Removed TestFromCloud test class Ref: DGR-151 Link: https://istari.atlassian.net/browse/DGR-151
Contributor
Author
DGR-151: Removed Dgraph Cloud ReferencesChanges MadeDgraph Cloud no longer exists, so all cloud-specific functionality and documentation has been removed from pydgraph. Code ChangesRemoved cloud-specific methods:
Cleanup:
Documentation ChangesREADME.md:
Examples:
Deprecated Notebooks:
These two notebooks cannot be easily updated as they rely heavily on cloud-specific features (Lambda functions, cloud authentication, cerebro API) that no longer exist. Migration Path for UsersUsers who were using Before (deprecated): client_stub = pydgraph.DgraphClientStub.from_cloud(
"endpoint.grpc.cloud.dgraph.io:443",
"api-key"
)After (recommended): # Using connection string
client = pydgraph.open("dgraph://endpoint:9080")
# Or using standard stub with SSL
creds = grpc.ssl_channel_credentials()
client_stub = pydgraph.DgraphClientStub("endpoint:443", creds)
client = pydgraph.DgraphClient(client_stub)Testing
|
Contributor
|
@mlwelles I'm wondering if it wouldn't be better to mark the Dgraph Cloud stuff as deprecated. People may still have references to that code in their applications, and running checkers like I agree with removing it from the docs, tho. @raphael-istari your thoughts? |
Restores the from_cloud() and parse_host() static methods to both DgraphClientStub and AsyncDgraphClientStub classes that were removed in commit 964e27f. These methods are now marked as deprecated using Python's warnings module with clear migration guidance: - Deprecated in version 25.1.0 - Planned removal in version 26.0.0 - Docstrings include migration examples showing how to use standard grpc.ssl_channel_credentials() and grpc.composite_channel_credentials() This maintains backward compatibility for existing code while guiding users to migrate to the standard SSL credential pattern.
Update CHANGELOG.md to accurately reflect that from_cloud() and parse_host() methods are deprecated rather than removed. Methods remain functional with deprecation warnings and will be removed in version 26.0.0.
Adds a proper gRPC health check to ensure the service is ready before running tests, addressing flaky test failures in CI. Changes: - Created scripts/wait_for_grpc_ready.py with gRPC channel readiness check - Uses exponential backoff pattern (1, 2, 3, 4 seconds repeating) - Integrated into scripts/local-test.sh after HTTP health check - Uses uv run when available to ensure grpc dependency is accessible - Replaces crude "sleep 5" delay with actual service verification This should reduce intermittent failures in CI tests against DGraph HEAD where the gRPC service isn't fully initialized despite HTTP endpoint being healthy.
Add retry logic and better error handling for test setup operations to handle timing issues with cluster initialization. Changes: - Add retry logic to drop_all() and set_schema() helper functions to handle UNAVAILABLE errors during cluster startup - Improve login retry logic to handle both "user not found" and "invalid username or password" errors during ACL initialization - Use explicit IPv4 addresses (127.0.0.1) instead of localhost to avoid IPv6 resolution issues - Add -v flag to stopCluster() to remove Docker volumes between test runs, ensuring clean state These improvements make the test suite more resilient to timing issues when the Dgraph cluster is initializing.
The wait_for_grpc_ready.py script is no longer needed because the retry logic in test helpers (drop_all and set_schema) already handles UNAVAILABLE errors during cluster initialization. Verified all 125 tests pass without the gRPC health check (93.01s). The HTTP health check combined with test helper retries is sufficient.
mlwelles
force-pushed
the
mlwelles/modernize-project-and-improve-typing
branch
from
06e1b61 to
e232eb2
Compare
- Combine nested if statements (SIM102) for cleaner code - Add explicit fallback raise after retry loops (RET503) to satisfy type checker even though the code path is unreachable
The is_aborted_error function was using private gRPC APIs (grpc._channel._Rendezvous and grpc._channel._InactiveRpcError) which were removed/moved in grpcio 1.76.0. This fix uses duck typing to check if the error has a code() method, which works for both sync (grpc.RpcError) and async (grpc.aio.AioRpcError) error types, and is compatible with all grpcio versions. Also added unit tests for is_aborted_error function. Original-Author: Shaun Patterson <shaunpatterson@gmail.com> Cherry-picked-from: PR #286
Resolved conflicts: - __init__.py: Kept explicit imports and __all__, added retry utilities - util.py: Kept type annotations for is_aborted_error
- Add full type annotations to retry.py (context managers, generators, functions) - Convert test assertions to pytest-style in test_retry.py - Fix duplicate test methods in test_util.py from merge conflict - Use Optional[T] instead of T | None for Python 3.9 compatibility - Disable pep585-upgrade hook (auto-upgrades to 3.10+ syntax) - Add mypy override for pydgraph.client arg-type (ty handles correctly) - Add type: ignore comments for duck-typed gRPC error handling - Remove unused type: ignore comments flagged by ty
mlwelles
force-pushed
the
mlwelles/modernize-project-and-improve-typing
branch
from
51313ef to
776784a
Compare
matthewmcneely
requested changes
Jan 16, 2026
- Change gRPC version check from RuntimeError to RuntimeWarning to avoid breaking apps with older gRPC versions - Add ## help strings to deps-uv and deps-trunk Makefile targets - Standardize deps-* help strings to note INSTALL_MISSING_TOOLS behavior
grpcio-tools generates code that raises RuntimeError for version mismatches, which breaks apps using older grpc versions. The protogen script now patches the generated api_pb2_grpc.py to use warnings.warn() instead. See: #282 (comment)
matthewmcneely
approved these changes
Jan 16, 2026
The branch protection requires "Trunk Code Quality / Check" but the workflow was reporting "Trunk Code Quality Checks / Check" (extra "s"). Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
mlwelles
added a commit
that referenced
this pull request
Feb 11, 2026
#282) ## Summary Comprehensive modernization of the pydgraph project with improved tooling, strict type safety, enhanced CI/CD infrastructure, and better developer experience. This PR adds type annotations to the entire codebase, migrates to modern Python tooling (uv, ruff, ty), establishes robust testing infrastructure, and provides comprehensive contribution guidelines. **Review Notes:** While the overall diff is large (23k+ additions, 20k- deletions across 68 files), the 86 commits are organized logically around specific themes. The bulk of changes fall into five categories: type annotations, tooling setup, CI/CD infrastructure, documentation, and code quality fixes. Each commit follows conventional commit format with clear descriptions. ## Related Issues - [DGR-137](https://istari.atlassian.net/browse/DGR-137) - Add type annotations to pydgraph - [DGR-138](https://istari.atlassian.net/browse/DGR-138) - Migrate to modern Python tooling (uv, ruff) - [DGR-139](https://istari.atlassian.net/browse/DGR-139) - Establish comprehensive CI/CD infrastructure - [DGR-140](https://istari.atlassian.net/browse/DGR-140) - Add Docker-based testing infrastructure - [DGR-141](https://istari.atlassian.net/browse/DGR-141) - Update Python version support (drop 3.7/3.8, add 3.13/3.14) - [DGR-142](https://istari.atlassian.net/browse/DGR-142) - Add comprehensive documentation (CONTRIBUTING.md) - [DGR-143](https://istari.atlassian.net/browse/DGR-143) - Fix code quality issues and linting errors - [DGR-144](https://istari.atlassian.net/browse/DGR-144) - Update protobuf generation with type stubs - [DGR-151](https://istari.atlassian.net/browse/DGR-151) - Deprecate Dgraph Cloud references ## Key Changes ### 🛠️ Project Tooling & Setup - **Migrated to [uv](https://docs.astral.sh/uv/)** - Modern Python package manager (10-100x faster than pip) - Replaced traditional pip/virtualenv workflow - Added `uv.lock` for reproducible builds - All scripts and workflows use `uv run` and `uv sync` - **Added comprehensive Makefile** with intuitive targets: - `make setup` - One-command project setup (installs tools, hooks, syncs deps) - `make check` - Runs all pre-commit hooks (ruff, mypy, ty, trunk, shellcheck) - `make test` - Runs test suite with Docker infrastructure - `make protogen` - Regenerates protobuf files with mypy stubs - `make build` / `make publish` - Build and publish releases - Automatic dependency checking with `INSTALL_MISSING_TOOLS=true` flag - Supports uv, trunk, and Docker installation - **Pre-commit hooks** - 24 hooks across 8 categories run on every commit: - File validation (large files, YAML/TOML/JSON syntax, EOF/whitespace) - Shell linting (shellcheck) - YAML formatting (yamlfmt) - Python quality (ruff lint + format, blanket noqa checks, type annotations enforced) - Type checking (mypy, ty) - Trunk integration (trunk fmt, trunk check) ### ✨ Type Safety & Code Quality - **Added comprehensive type annotations** to all Python files (23+ files, 300+ functions) - Core modules: `client.py`, `async_client.py`, `txn.py`, `async_txn.py` - Stub modules: `client_stub.py`, `async_client_stub.py` - Utilities: `util.py`, `errors.py`, `convert.py` - All test files (15 files) - Build scripts: `protogen.py` - **Enabled strict mypy type checking** - `disallow_untyped_defs = true` - `disallow_incomplete_defs = true` - `check_untyped_defs = true` - All functions have proper type signatures - **Modern type annotation syntax** - `from __future__ import annotations` for forward references - PEP 604 union syntax: `X | None` instead of `Optional[X]` - `-> None` for all void functions - **Added [ruff](https://docs.astral.sh/ruff/)** - Extremely fast Python linter/formatter (10-100x faster than Flake8/Black) - Comprehensive ruleset: 800+ rules across 20+ categories - Configured in `pyproject.toml` with sensible defaults - Automatically fixes issues in pre-commit - **Added [ty](https://docs.astral.sh/ty/)** - Modern type checker with advanced error diagnostics - Runs alongside mypy for comprehensive type coverage - Better error messages than traditional type checkers ### 🧹 Code Quality Improvements - **Fixed all ruff linting issues** across source and test files: - **Builtin shadowing** (A001/A002/A004): Added strategic `# noqa` for intentional public API (`open`, `ConnectionError`) - **Exception handling** (TRY200/TRY300/TRY301): Refactored for correctness - Moved return statements outside try blocks - Use bare `raise` to preserve stack traces - Moved validation outside exception handlers - **Security** (S311): Replaced `random.choice()` → `secrets.choice()` - **Code patterns** (SIM105): Use `contextlib.suppress()` for clarity - **Improved test assertions** (PT011/PT017) - Added `match` parameters to 20+ `pytest.raises` calls - Replaced try-except-assert with proper `pytest.raises` - **Removed Python 2 compatibility code** - Removed urlparse fallback imports - Removed basestring and long type checks - Simplified string type checking - **Fixed code quality issues** - Fixed undefined variable references - Removed wildcard imports, added explicit `__all__` - Exported `pydgraph.open` function - Fixed variable shadowing in exception handlers ### 🔄 CI/CD Infrastructure **Separated workflows for better organization:** 1. **ci-pydgraph-tests.yml** - Matrix testing across Python versions - **Test (Python 3.9-3.14 / DGraph Latest)** - 6 parallel jobs testing against latest Dgraph release - **Test (Python 3.9-3.14 / DGraph HEAD)** - 6 parallel jobs testing against Dgraph main branch - Uses `setup-python-and-tooling` composite action - Explicit `make setup` and `make sync` steps - Tests 12 Python/Dgraph combinations in parallel 2. **ci-pydgraph-code-quality.yml** - Code quality checks - Runs on Python 3.13 (canonical development version) - Protobuf verification (ensures generated files are current) - `make check` - All pre-commit hooks (ruff, mypy, ty, trunk, shellcheck) - Runs with `SKIP=trunk-check,trunk-fmt` to avoid duplication with trunk workflow 3. **ci-pydgraph-trunk.yml** - Trunk code quality checks - Uses dgraph-io/.github reusable trunk workflow - Provides inline PR comments for issues - Separate from other checks for clear separation of concerns 4. **cd-pydgraph.yml** - Release workflow - Workflow dispatch for manual releases - Runs full test suite before publishing - PyPI publishing with UV_PUBLISH_USERNAME and UV_PUBLISH_PASSWORD - Uses `uv version` and `uv publish` commands **Shared infrastructure:** - **setup-python-and-tooling** composite action (renamed from setup-runner) - Sets up specified Python version with caching - Installs uv package manager - Used by all workflows for consistency ### 🐳 Testing Infrastructure - **Docker-based test setup** via `scripts/local-test.sh` - Automatic Dgraph cluster startup/teardown - Dynamic port allocation prevents conflicts - Isolated test environments - Supports `DGRAPH_IMAGE_TAG` for testing different Dgraph versions - **Matrix testing** - All tests run on Python 3.9, 3.10, 3.11, 3.12, 3.13, 3.14 - **Test results**: ✅ 125 passed - **Docker dependency checking** in Makefile - Validates Docker 20.10.0+ and Docker Compose v2 - Auto-install support on macOS and Linux ### 🔄 Dependency Updates - **Flexible protobuf support**: `>=4.23.0,<7.0.0` - Supports protobuf 4.x, 5.x, and 6.x - Users can pin to specific versions for compatibility - Modern environments (Python 3.13+) use 6.x by default - **Updated grpcio**: `>=1.65.0,<2.0.0` (loosened from strict 1.65.1) - Addresses build issues on modern systems - Older versions fail to compile with recent Xcode/gcc - **Added comprehensive dev dependencies**: - Type checking: `mypy>=1.14.1`, `ty>=0.0.8`, `grpc-stubs>=1.53.0.6` - Type stubs: `types-grpcio>=1.0.0`, `types-protobuf>=6.32.1` - Linting: `ruff>=0.8.4` - Testing: `pytest>=8.3.3`, `pytest-asyncio>=0.23.0` - Tooling: `pre-commit>=3.5.0`, `shellcheck-py>=0.10.0.1` ### ☁️ Dgraph Cloud Deprecation ([DGR-151](https://istari.atlassian.net/browse/DGR-151)) - **Deprecated cloud-specific functionality** (deprecated in 25.1.0, removal planned for 26.0.0): - `from_cloud()` static methods in `DgraphClientStub` and `AsyncDgraphClientStub` - `parse_host()` helper methods - Methods restored with deprecation warnings and migration guidance - **Updated documentation**: - Removed "Connecting To Dgraph Cloud" section from README.md - Removed `apikey` parameter from connection string documentation - Removed cloud-specific connection string examples - **Simplified examples**: - Updated `examples/embeddings/computeEmbeddings.py` to use standard connection - Updated RAG notebook to use standard Dgraph connection - Added deprecation warnings to cloud-dependent notebooks (`dgraph-episode1.ipynb`, `dgraph-ai-classification.ipynb`) - **Rationale**: Dgraph Cloud no longer exists, so cloud-specific features (Lambda functions, cerebro API, cloud authentication) are no longer applicable - **Migration path**: Users should migrate to standard `grpc.ssl_channel_credentials()` pattern (see method docstrings for examples) ### 🐍 Python Version Support - **Added support**: Python 3.13 and 3.14 - **Dropped support**: Python 3.7 and 3.8 - Both versions reached end of life - No longer supported by critical dependencies - Required for modern type annotation features - **New minimum**: Python 3.9 - **Development version**: Python 3.13 (for protobuf generation) - **CI testing**: All versions 3.9-3.14 tested in parallel ### 📚 Documentation - **Added CONTRIBUTING.md** - Comprehensive contribution guide - Development setup with `make setup` - Code style and standards (SPDX headers, type hints, ruff formatting) - Testing procedures and infrastructure - PR requirements and conventional commits - Makefile command reference - Protobuf generation requirements - grpcio version compatibility notes - **Added CODE_OF_CONDUCT.md** - Contributor Covenant - **Updated README.md** - Simplified to focus on usage - Links to CONTRIBUTING.md for development - Updated examples to use `uv run python` - Removed duplicate development content - Removed Dgraph Cloud references - **Updated example READMEs** - All use modern `uv run` workflow - **Updated PUBLISHING.md** - New protogen command reference ### 📦 Project Metadata - **Updated license to SPDX format**: `license = "Apache-2.0"` - Fixes setuptools deprecation warning - Removed deprecated license classifier - **Updated author/maintainer**: Istari Digital, Inc. - Updated `__author__` and `__maintainer__` in 18+ Python files - Updated email to dgraph-admin@istaridigital.com - **Updated Homepage URL**: https://github.com/dgraph-io/pydgraph - **Updated classifiers**: Python 3.9-3.14 - **Updated requires-python**: `>=3.9` ### 🔧 Protobuf Generation - **Enhanced `scripts/protogen.py`**: - Version validation (requires Python 3.13+, grpcio-tools 1.66.2+) - Generates mypy type stubs (.pyi files) - Explicit error messages for version mismatches - Documents canonical development environment - **Generated files updated**: - `api_pb2.py`, `api_pb2_grpc.py` - Protobuf implementations - `api_pb2.pyi`, `api_pb2_grpc.pyi` - Type stubs for better IDE support - **CI verification**: Code quality workflow ensures generated files are current ## Breaking Changes - **Minimum Python version increased from 3.7 to 3.9** - Python 3.7 and 3.8 reached end of life - Required for dependency compatibility and modern typing features ## Deprecations - **Dgraph Cloud-specific functionality** ([DGR-151](https://istari.atlassian.net/browse/DGR-151)) - `DgraphClientStub.from_cloud()` method deprecated (removal planned for v26.0.0) - `AsyncDgraphClientStub.from_cloud()` method deprecated (removal planned for v26.0.0) - `DgraphClientStub.parse_host()` method deprecated (removal planned for v26.0.0) - `AsyncDgraphClientStub.parse_host()` method deprecated (removal planned for v26.0.0) - Methods still functional but emit `DeprecationWarning` - Migration guidance provided in method docstrings - Users should migrate to standard connection methods with `grpc.ssl_channel_credentials()` ## Backwards Compatibility - **Protobuf version flexibility**: Wide version support (4.23.0 - 6.x) ensures compatibility - Users can pin to older versions if needed: ```bash pip install pydgraph "protobuf>=4.23.0,<5.0.0" # protobuf 4.x pip install pydgraph "protobuf>=5.0.0,<6.0.0" # protobuf 5.x ``` - **grpcio flexibility**: Supports 1.65.0+ for broad compatibility - **Deprecated methods remain functional**: Cloud-specific methods still work but emit warnings - **No other API changes**: All existing public APIs remain unchanged ## Benefits - **Type Safety**: Comprehensive annotations catch errors at development time - **Better IDE Support**: Full autocomplete and type hints in all editors - **Maintainability**: Type signatures serve as inline documentation - **Code Quality**: Strict linting and formatting ensure consistency - **Modern Tooling**: uv provides 10-100x faster dependency management - **Test Isolation**: Docker-based tests prevent port conflicts - **Comprehensive CI/CD**: Automated quality checks on every PR across all Python versions - **Developer Experience**: One-command setup (`make setup`), clear documentation - **Dependency Flexibility**: Wide protobuf/grpcio support ensures compatibility - **Graceful Deprecation**: Cloud functionality deprecated with clear migration path ## Testing - ✅ All pre-commit hooks pass (ruff, mypy, ty, trunk, shellcheck, yamlfmt) - ✅ All tests pass: 125 passed in ~80s - ✅ Tests verified on Python 3.9, 3.10, 3.11, 3.12, 3.13, 3.14 - ✅ Protobuf generation works with Python 3.13 - ✅ Type stubs generated successfully - ✅ Docker test infrastructure verified on macOS and Ubuntu - ✅ Deprecation warnings verified for cloud methods - ✅ CI workflows passing: - CodeQL security analysis - Code quality checks (pre-commit hooks) - Matrix tests (12 Python/Dgraph combinations) - Trunk checks ## Tool References - [uv](https://docs.astral.sh/uv/) - Python package manager | [GitHub](https://github.com/astral-sh/uv) - [ruff](https://docs.astral.sh/ruff/) - Python linter/formatter | [GitHub](https://github.com/astral-sh/ruff) - [ty](https://docs.astral.sh/ty/) - Type checker | [Announcement](https://astral.sh/blog/ty) - [trunk](https://docs.trunk.io/) - Code quality platform | [GitHub](https://github.com/trunk-io/trunk) - [mypy](https://mypy-lang.org/) - Static type checker --------- Co-authored-by: Matthew McNeely <matthew.mcneely@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comprehensive modernization of the pydgraph project with improved tooling, strict type safety, enhanced CI/CD infrastructure, and better developer experience. This PR adds type annotations to the entire codebase, migrates to modern Python tooling (uv, ruff, ty), establishes robust testing infrastructure, and provides comprehensive contribution guidelines.
Review Notes: While the overall diff is large (23k+ additions, 20k- deletions across 68 files), the 86 commits are organized logically around specific themes. The bulk of changes fall into five categories: type annotations, tooling setup, CI/CD infrastructure, documentation, and code quality fixes. Each commit follows conventional commit format with clear descriptions.
Key Changes
🛠️ Project Tooling & Setup
uv.lockfor reproducible buildsuv runanduv syncmake setup- One-command project setup (installs tools, hooks, syncs deps)make check- Runs all pre-commit hooks (ruff, mypy, ty, trunk, shellcheck)make test- Runs test suite with Docker infrastructuremake protogen- Regenerates protobuf files with mypy stubsmake build/make publish- Build and publish releasesINSTALL_MISSING_TOOLS=trueflag✨ Type Safety & Code Quality
client.py,async_client.py,txn.py,async_txn.pyclient_stub.py,async_client_stub.pyutil.py,errors.py,convert.pyprotogen.pydisallow_untyped_defs = truedisallow_incomplete_defs = truecheck_untyped_defs = truefrom __future__ import annotationsfor forward referencesX | Noneinstead ofOptional[X]-> Nonefor all void functionspyproject.tomlwith sensible defaults🧹 Code Quality Improvements
# noqafor intentional public API (open,ConnectionError)raiseto preserve stack tracesrandom.choice()→secrets.choice()contextlib.suppress()for claritymatchparameters to 20+pytest.raisescallspytest.raises__all__pydgraph.openfunction🔄 CI/CD Infrastructure
Separated workflows for better organization:
ci-pydgraph-tests.yml - Matrix testing across Python versions
setup-python-and-toolingcomposite actionmake setupandmake syncstepsci-pydgraph-code-quality.yml - Code quality checks
make check- All pre-commit hooks (ruff, mypy, ty, trunk, shellcheck)SKIP=trunk-check,trunk-fmtto avoid duplication with trunk workflowci-pydgraph-trunk.yml - Trunk code quality checks
cd-pydgraph.yml - Release workflow
uv versionanduv publishcommandsShared infrastructure:
🐳 Testing Infrastructure
scripts/local-test.shDGRAPH_IMAGE_TAGfor testing different Dgraph versions🔄 Dependency Updates
>=4.23.0,<7.0.0>=1.65.0,<2.0.0(loosened from strict 1.65.1)mypy>=1.14.1,ty>=0.0.8,grpc-stubs>=1.53.0.6types-grpcio>=1.0.0,types-protobuf>=6.32.1ruff>=0.8.4pytest>=8.3.3,pytest-asyncio>=0.23.0pre-commit>=3.5.0,shellcheck-py>=0.10.0.1☁️ Dgraph Cloud Deprecation (DGR-151)
from_cloud()static methods inDgraphClientStubandAsyncDgraphClientStubparse_host()helper methodsapikeyparameter from connection string documentationexamples/embeddings/computeEmbeddings.pyto use standard connectiondgraph-episode1.ipynb,dgraph-ai-classification.ipynb)grpc.ssl_channel_credentials()pattern (see method docstrings for examples)🐍 Python Version Support
📚 Documentation
make setupuv run pythonuv runworkflow📦 Project Metadata
license = "Apache-2.0"__author__and__maintainer__in 18+ Python files>=3.9🔧 Protobuf Generation
scripts/protogen.py:api_pb2.py,api_pb2_grpc.py- Protobuf implementationsapi_pb2.pyi,api_pb2_grpc.pyi- Type stubs for better IDE supportBreaking Changes
Deprecations
DgraphClientStub.from_cloud()method deprecated (removal planned for v26.0.0)AsyncDgraphClientStub.from_cloud()method deprecated (removal planned for v26.0.0)DgraphClientStub.parse_host()method deprecated (removal planned for v26.0.0)AsyncDgraphClientStub.parse_host()method deprecated (removal planned for v26.0.0)DeprecationWarninggrpc.ssl_channel_credentials()Backwards Compatibility
Benefits
make setup), clear documentationTesting
Tool References