Bump minimal supported Python version to py3.11

[xsedla1o]

Python 3.9 has reached EOL and Python 3.10 EOL is approaching, so this branch bumps DP3’s supported/runtime baseline to Python 3.11 and modernizes the codebase/tooling accordingly.

Key changes

• Require Python 3.11 in packaging, Ruff/Black targets, Docker images, CI docs deploy, README, and docs.
• Apply Ruff/Black modernization for Python 3.11, including PEP 604 union types and related typing/import cleanup.
• Fix Python 3.11/runtime typing issues around dynamic API entity ID validation and MACAddress.
• Replace deprecated Pydantic APIs with current equivalents.
• Preserve /entity/{etype}/{eid}/set/{attr} behavior by explicitly parsing JSON request bodies.
• Add small typing cleanups for config defaults and task queue pyright compatibility.

[Copilot]

Pull request overview

This PR raises DP3’s Python baseline to 3.11 and modernizes typing/tooling accordingly, including updates to runtime/API code paths that rely on Pydantic validation and datetime handling.

Changes:

• Bump declared/runtime Python version to 3.11 across packaging, tooling targets, Docker images, CI deploy, and docs.
• Modernize typing (PEP 604 unions, collections.abc imports), and update Pydantic v2 usage (model_validate, model_dump).
• Adjust API/runtime behavior around entity ID validation and JSON request body parsing for /entity/{etype}/{eid}/set/{attr}.

Reviewed changes

Copilot reviewed 58 out of 63 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/test_example/dps_gen.py	Switch to datetime.UTC for UTC timestamps in test generator.
tests/test_example/dps_gen_realtime.py	Switch to datetime.UTC and remove local UTC alias.
tests/test_common/test_types.py	Update UTC expectations to use datetime.UTC.
tests/test_common/test_snapshots.py	Typing modernizations (PEP 604, collections.abc.Callable), UTC import cleanup.
tests/test_common/test_module_testing.py	Use datetime.UTC instead of project UTC alias.
tests/test_common/test_magic.py	Use datetime.UTC in expected datetime values.
tests/test_api/test_telemetry.py	Use datetime.UTC in API tests.
tests/test_api/test_snapshots.py	Use datetime.UTC in API tests.
tests/test_api/test_raw.py	Use datetime.UTC in API tests.
tests/test_api/test_get_entity_eid_data.py	Use datetime.UTC in API tests.
tests/test_api/test_01_datapoints.py	Use datetime.UTC in API tests.
tests/test_api/common.py	Prefer collections.abc.Callable over typing.Callable.
requirements.txt	Relax some patch pins while keeping compatible minor versions.
requirements.scripts.txt	Update pandas major line (2.2).
README.md	Document Python 3.11+ prerequisites.
pyproject.toml	Set requires-python >=3.11 and update Black/Ruff targets to py311.
dp3/testing/registrar.py	Typing modernizations in test registrar APIs.
dp3/testing/config.py	Typing modernizations (PEP 604 optionals).
dp3/testing/case.py	Typing modernizations; use datetime.UTC.
dp3/template/app/docker/python/Dockerfile	Bump template Python image to 3.11-slim.
dp3/task_processing/task_queue.py	Typing updates, thread liveness fixes, small robustness tweaks.
dp3/task_processing/task_hooks.py	Prefer collections.abc.Callable.
dp3/task_processing/task_executor.py	Prefer collections.abc.Callable.
dp3/snapshots/snapshot_hooks.py	Typing modernizations for hook signatures.
dp3/snapshots/snapshooter.py	Typing modernizations; migrate to model_dump; zip strictness.
dp3/scripts/dummy_sender.py	Switch to datetime.UTC and remove local UTC alias.
dp3/scripts/datapoint_log_converter.py	Typing import cleanup (collections.abc.Callable).
dp3/scripts/add_hashes.py	Replace deprecated Pydantic parse_obj with model_validate.
dp3/history_management/telemetry.py	Use datetime.UTC; typing modernizations.
dp3/history_management/history_manager.py	Migrate Pydantic extra handling to ConfigDict; typing modernizations.
dp3/database/snapshots.py	Typing modernizations (PEP 604 unions).
dp3/database/schema_cleaner.py	Use datetime.UTC; typing import cleanup.
dp3/database/magic.py	Use datetime.UTC; typing modernizations for helper return types.
dp3/database/database.py	Use datetime.UTC; typing modernizations; zip strictness.
dp3/database/config.py	Typing modernizations for discriminated union config.
dp3/core/updater.py	Use datetime.UTC; typing import cleanup.
dp3/core/link_manager.py	Use datetime.UTC.
dp3/core/collector.py	Use datetime.UTC.
dp3/common/utils.py	Typing modernizations for duration parsing signature.
dp3/common/types.py	Use datetime.UTC; PEP 604 typing cleanup.
dp3/common/task.py	Typing modernizations for validator signatures and fields.
dp3/common/scheduler.py	Typing modernizations for scheduler API.
dp3/common/mac_address.py	Add __future__.annotations; modernize Pydantic core schema typing.
dp3/common/entityspec.py	Typing modernizations for discriminated union fields.
dp3/common/datatype.py	Typing modernizations for primitive types and aliases.
dp3/common/datapoint.py	Typing modernizations; PEP 604 unions for aliases/fields.
dp3/common/control.py	Prefer collections.abc.Callable.
dp3/common/config.py	Pydantic ConfigDict migration; improve typing for HierarchicalDict.get.
dp3/common/callback_registrar.py	Typing modernizations for hook signatures and scheduler registration.
dp3/common/attrspec.py	Typing modernizations (PEP 604 unions) for config models and aliases.
dp3/bin/shcmd/entity/common.py	Typing modernizations for optional return types/params.
dp3/bin/shcmd/entity/init.py	Typing modernizations for return types.
dp3/bin/shcmd/common.py	Typing modernizations for API client helpers and completion loading.
dp3/bin/check.py	Use zip(..., strict=False) for safer zips.
dp3/api/routers/telemetry.py	Typing modernizations for optional query params.
dp3/api/routers/entity.py	Update entity ID parsing typing; explicitly parse JSON body for set-attr endpoint; use model_dump.
dp3/api/internal/models.py	Rework dynamic EntityId model generation and TypeAdapter union typing.
dp3/api/internal/entity_response_models.py	Typing modernizations for response model aliases.
dp3/api/internal/config.py	Replace deprecated Pydantic parse_obj with model_validate.
docs/howto/get-started.md	Document Python 3.11+ prerequisites.
docs/howto/develop-dp3.md	Document Python 3.11+ prerequisites.
docker/python/Dockerfile	Bump runtime Python image to 3.11-slim.
.github/workflows/deploy.yml	Use Python 3.11 for docs deploy workflow.

Comments suppressed due to low confidence (1)

dp3/snapshots/snapshooter.py:582

• master_record[datapoint.attr].append() is called without an argument, which will raise TypeError and break snapshot generation. This likely should append the newly created dp_dict (and should preserve the same structure used when initializing the list in the else branch).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dbnk0]

LGTM