Skip to content

Conversation

@pinglin
Copy link
Member

@pinglin pinglin commented Oct 30, 2025

Because

  • The protobuf definitions were refactored in CE, renaming Catalog to KnowledgeBase and restructuring the artifact service APIs
  • Credit and subscription management endpoints were removed from the CE version of the mgmt API
  • Python 3.13 support is needed for the SDK
  • Multiple dependency conflicts existed between newer packages required for Python 3.13 and older transitive dependencies

This commit

  • Refactors ArtifactClient (instill/clients/artifact.py):

    • Renames all catalog methods to knowledge_base methods
    • Updates imports to reflect new protobuf file structure (artifact_pb2file_pb2, knowledge_base_pb2)
    • Adds new methods: list_knowledge_base_runs(), get_chunk()
    • Removes deprecated methods: process_catalog_files(), question_answering(), get_file_catalog()
    • Updates search_chunks() with new parameters: chunk_type, file_media_type, file_ids, tags
    • Fixes mypy type errors by properly constructing protobuf requests instead of using dict unpacking
    • Adds type ignore comments for protobuf enum assignments
  • Updates MgmtClient (instill/clients/mgmt.py):

    • Removes CE-specific credit endpoints: get_remaining_credit(), list_credit_consumption_chart_records()
    • Removes CE-specific subscription endpoints: get_authenticated_subscription(), get_organization_subscription()
  • Fixes file processing utility (instill/utils/process_file.py):

    • Updates imports from artifact_pb2 to file_pb2 for File message
    • Changes enum handling from string format ("FILE_TYPE_TEXT") to protobuf enum constants (file_interface.File.TYPE_TEXT)
    • Fixes field name from name to filename in File construction
    • Returns proper enum values instead of strings from get_file_type()
  • Updates tests (instill/tests/test_utils.py):

    • Adds file_interface import for protobuf enums
    • Updates all assertions to use enum constants instead of string values
    • Fixes field references from result.name to result.filename
    • Separates CSV type testing from TEXT type (CSV has its own TYPE_CSV)
    • Adds pylint disable comment for protobuf module false positives
  • Upgrades dependencies for Python 3.13 support (pyproject.toml):

    • Updates Python version constraint: >=3.9,<3.13>=3.9,<3.14
    • Updates core dependencies:
      • grpcio: ^1.59.0^1.68.0 (Python 3.13 support)
      • grpcio-tools: 1.62.0^1.68.0 (Python 3.13 build support)
      • protobuf: 4.25.8^5.28.0 (required by grpcio 1.68+)
      • types-protobuf: ^4.24.0.1^5.28.0
      • numpy: ^1.21.0>=1.26,<1.28 (Python 3.13 support while avoiding scipy/cupy conflicts)
      • pillow: ^10.1.0^11.0.0
      • pyinstaller: 6.0.0^6.16.0
      • google-api-core: ^2.11.1{version = "^2.22.0", extras = ["grpc"]}
      • ray: 2.47.0{version = "^2.47.0", extras = ["default"]}
    • Restricts incompatible macOS dev dependencies to Python < 3.13:
      • MacFSEvents = {version = "*", platform = "darwin", python = "<3.13"}
      • sniffer = {version = "*", python = "<3.13"}
    • Explicitly upgrades opentelemetry packages to resolve protobuf 5.x conflicts:
      • opentelemetry-proto = "^1.29.0"
      • opentelemetry-exporter-otlp = "^1.29.0"
      • opentelemetry-exporter-otlp-proto-common = "^1.29.0"
      • opentelemetry-exporter-otlp-proto-grpc = "^1.29.0"
      • opentelemetry-exporter-otlp-proto-http = "^1.29.0"
  • Resolves all dependency conflicts:

    • Eliminates numpy version warnings from scipy/cupy by using numpy 1.26-1.27
    • Fixes opentelemetry-proto protobuf <5.0 requirement by upgrading to 1.29+
    • Ensures all opentelemetry exporters are compatible versions

Testing

  • All mypy type checks pass
  • All pylint checks pass (excluding pre-existing warnings)
  • File type enum handling verified through test updates
  • Dependency installation completes without conflicts

Breaking Changes

  • Method names in ArtifactClient changed from *_catalog* to *_knowledge_base*
  • Removed methods: process_catalog_files(), question_answering(), get_file_catalog()
  • Removed methods in MgmtClient: get_remaining_credit(), list_credit_consumption_chart_records(), get_authenticated_subscription(), get_organization_subscription()
  • process_file() utility now returns protobuf enum values instead of strings for file types

@pinglin pinglin force-pushed the pinglin/refactor-python-sdk-ce-protobuf-python313 branch 8 times, most recently from 9298430 to 9a44f12 Compare October 30, 2025 12:22
@pinglin pinglin force-pushed the pinglin/refactor-python-sdk-ce-protobuf-python313 branch from 9a44f12 to 48c9bde Compare October 30, 2025 12:26
@pinglin pinglin merged commit a6f16a8 into main Oct 30, 2025
10 checks passed
@pinglin pinglin deleted the pinglin/refactor-python-sdk-ce-protobuf-python313 branch October 30, 2025 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants