Skip to content

build(application): upgrade azure_iot_operations binaries in 500 applications#519

Merged
kgmwang1 merged 43 commits into
mainfrom
feat/298-aio-sdk-telemetry-upgrade
Jun 25, 2026
Merged

build(application): upgrade azure_iot_operations binaries in 500 applications#519
kgmwang1 merged 43 commits into
mainfrom
feat/298-aio-sdk-telemetry-upgrade

Conversation

@kgmwang1

@kgmwang1 kgmwang1 commented May 15, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR upgrades Azure IoT Operations dependencies across 500-application services to current crates.io releases, refreshes related lockfiles and runtime code paths, and hardens Rust CI behavior for private-registry scenarios.

Why

Move off older aio-sdks-pinned crate versions in application services.
Align service dependency versions and lockfiles for consistent builds.
Reduce CI fragility when private registry auth is unavailable.
Carry forward security/build fixes from dependency refreshes.

What Changed

Upgraded/migrated Azure IoT Operations Rust dependencies across application services:
azure_iot_operations_mqtt moved to 1.0.2 in multiple services.
azure_iot_operations_protocol moved to 1.0.0 where required.
azure_iot_operations_services standardized to 1.2.0 with state_store.
Refreshed Cargo lockfiles and selected Docker/runtime code for affected services.
Updated CI workflow behavior in rust-tests.yml :
Handles aio-sdks token-based auth more explicitly.
Skips token-required coverage jobs when token is missing.
Uses cargo llvm-cov --locked for reproducibility.

API Breaking changes resolved:

  • SessionConnectionMonitor -> SessionMonitor
  • publish -> publish_qos1
  • Adopted TopicName, TopicFilter, SubscribeProperties, and PublishProperties
  • message.topic -> message.topic_name

Additional note for 507:

Removed its direct azure_iot_operations_protocol dependency.

Closes #298

@kgmwang1 kgmwang1 changed the title build(application): upgrade mqtt dependency in http connector build(application): upgrade azure_iot_operations binaries in 500 applications May 15, 2026
@codecov-commenter

codecov-commenter commented May 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 97 lines in your changes missing coverage. Please review.
✅ Project coverage is 31.80%. Comparing base (5c30b25) to head (a771adf).

Files with missing lines Patch % Lines
...i-inference/services/ai-edge-inference/src/mqtt.rs 0.00% 26 Missing ⚠️
...rvices/mqtt-otel-trace-exporter/src/mqtt_client.rs 0.00% 24 Missing ⚠️
...tp-connector/services/broker/src/mqtt_publisher.rs 0.00% 14 Missing ⚠️
...services/media-capture-service/src/mqtt_handler.rs 0.00% 13 Missing ⚠️
...ust-http-connector/services/subscriber/src/main.rs 0.00% 10 Missing ⚠️
...ion/501-rust-telemetry/services/sender/src/main.rs 0.00% 4 Missing ⚠️
...n/501-rust-telemetry/services/receiver/src/main.rs 0.00% 3 Missing ⚠️
...02-rust-http-connector/services/broker/src/main.rs 0.00% 2 Missing ⚠️
...tp-connector/services/broker/src/uptime_monitor.rs 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #519      +/-   ##
==========================================
- Coverage   32.09%   31.80%   -0.30%     
==========================================
  Files          40       40              
  Lines        5960     6015      +55     
==========================================
  Hits         1913     1913              
- Misses       4047     4102      +55     
Flag Coverage Δ
rust 31.80% <0.00%> (-0.30%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...vices/media-capture-service/src/video_processor.rs 12.92% <ø> (ø)
...tp-connector/services/broker/src/uptime_monitor.rs 0.00% <0.00%> (ø)
...02-rust-http-connector/services/broker/src/main.rs 0.00% <0.00%> (ø)
...n/501-rust-telemetry/services/receiver/src/main.rs 66.44% <0.00%> (ø)
...ion/501-rust-telemetry/services/sender/src/main.rs 47.45% <0.00%> (ø)
...ust-http-connector/services/subscriber/src/main.rs 0.00% <0.00%> (ø)
...services/media-capture-service/src/mqtt_handler.rs 0.00% <0.00%> (ø)
...tp-connector/services/broker/src/mqtt_publisher.rs 0.00% <0.00%> (ø)
...rvices/mqtt-otel-trace-exporter/src/mqtt_client.rs 0.00% <0.00%> (ø)
...i-inference/services/ai-edge-inference/src/mqtt.rs 3.11% <0.00%> (-0.13%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kgmwang1 kgmwang1 marked this pull request as ready for review May 22, 2026 21:30
@kgmwang1 kgmwang1 requested a review from a team as a code owner May 22, 2026 21:30
Comment thread .github/workflows/rust-tests.yml
Comment thread .github/workflows/rust-tests.yml Outdated
Comment thread src/500-application/502-rust-http-connector/services/broker/Cargo.toml Outdated
Comment thread src/500-application/507-ai-inference/services/ai-edge-inference-crate/Cargo.toml Outdated

@bindsi bindsi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review: AIO SDK upgrades (0.x→1.x) look correct across .NET and Rust crates. One observation:

reqwest downgrade (non-blocking): In 502-rust-http-connector/services/broker/Cargo.toml, reqwest is changed from 0.12 to 0.11.6. This is a semver downgrade — reqwest 0.12 has different APIs from 0.11 (e.g., the Body type and builder patterns changed). If the AIO SDK 1.0.2 pins to reqwest 0.11.x internally, this may be intentional for compatibility, but please confirm this doesn't regress any HTTP client functionality in the broker service. The same version (0.11.6) is also added as a new dep in media-capture-service.

@bindsi bindsi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated batch review: requesting changes for one blocking lockfile issue.

Comment thread .github/workflows/rust-tests.yml Outdated
kgmwang1 and others added 7 commits June 17, 2026 21:08
- update rust coverage step to skip upload when coverage.xml is absent
- ignore commented aio-sdks entries when deciding registry skip behavior
- bump reqwest manifest pins in broker and media-capture crates

🔒 - Generated by Copilot
🔧 - Generated by Copilot
…ures

- update dependencies for Azure IoT Operations SDK
- refactor MQTT publisher to use new session management
- implement enhanced message processing with topic filtering
- add pallet correlation enrichment functionality

🔧 - Generated by Copilot

Signed-off-by: Marcel Bindseil <marcelbindseil@gmail.com>
…de' into feat/298-aio-sdk-telemetry-upgrade

# Conflicts:
#	src/500-application/502-rust-http-connector/services/broker/Cargo.lock
#	src/500-application/502-rust-http-connector/services/broker/src/main.rs
#	src/500-application/502-rust-http-connector/services/broker/src/mqtt_publisher.rs
@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown

Dependency Review

The following issues were found:

  • ❌ 1 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 14 package(s) with unknown licenses.
  • ⚠️ 27 packages with OpenSSF Scorecard issues.

View full job summary

kgmwang1 and others added 2 commits June 20, 2026 17:42
- replace git SHA dependencies with published crates.io versions
- remove unused azure_iot_operations_aio_sdk dependency
- remove aio-sdks private registry from .cargo/config.toml files
- simplify Dockerfiles by removing registry injection blocks

📦 - Generated by Copilot
kgmwang1 and others added 6 commits June 22, 2026 21:01
…1/receiver

- receiver source imports azure_iot_operations_protocol but was missing from Cargo.toml
- now matches sender and other services that use protocol module

🔧 - Generated by Copilot
- regenerated lock file after adding azure_iot_operations_protocol dependency
- ensures reproducible builds with correct versions locked

🔧 - Generated by Copilot
- Format packages.lock.json files modified in this branch
- No functional changes, formatting only

🎨 - Generated by Copilot
- memmap2 0.9.10/0.9.11 has unsound unchecked pointer offset vulnerability
- Transitive dependency via candle-core 0.9.2 (used for AI inference)
- Cannot update without upstream candle release
- Documented in audit.toml alongside other transitive vulnerabilities

🔒 - Generated by Copilot
- Minor version updates from cargo update: autocfg 1.5.0 -> 1.5.1, http 1.4.0 -> 1.4.2, hyper 1.9.0 -> 1.10.1
- Ensures reproducible builds with up-to-date transitive dependencies

🔧 - Generated by Copilot
bindsi added a commit that referenced this pull request Jun 23, 2026
## Description

Patches two newly published Rust security advisories that are failing
the `Dependency Audit` (`cargo-audit`) CI job and blocking unrelated PRs
(for example #519) from merging. Both fixes are semver-compatible,
lockfile-only dependency bumps with no source or behavior changes.

- **quinn-proto** `0.11.14` → `0.11.15` in
`src/500-application/502-rust-http-connector/services/broker` to resolve
[RUSTSEC-2026-0185](https://rustsec.org/advisories/RUSTSEC-2026-0185.html)
— *Remote memory exhaustion from unbounded out-of-order stream
reassembly* (CVSS 7.5 HIGH, denial-of-service).
- **memmap2** `0.9.10` → `0.9.11` in
`src/500-application/507-ai-inference/services/ai-edge-inference` and
`ai-edge-inference-crate` to resolve
[RUSTSEC-2026-0186](https://rustsec.org/advisories/RUSTSEC-2026-0186.html)
— *Unchecked pointer offset* (unsound).

These were addressed by upgrading the affected dependencies to their
patched releases rather than allow-listing them in `.github/audit.toml`
/ `osv-scanner.toml`, because fixed versions are available and
semver-compatible.

## Related Issue

Relates to #519 (the cargo-audit failures were blocking that PR)

## Type of Change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Blueprint modification or addition
- [ ] Component modification or addition
- [ ] Documentation update
- [ ] CI/CD pipeline change
- [ ] Other (please describe):

## Implementation Details

- Ran `cargo update -p quinn-proto --precise 0.11.15` in the
`502-rust-http-connector` broker crate.
- Ran `cargo update -p memmap2 --precise 0.9.11` in both
`507-ai-inference` crates.
- Only `Cargo.lock` files changed; the affected entries are transitive
dependencies, so no `Cargo.toml` manifest edits were required.
- The diff is limited to the dependency version and checksum lines (6
lines across 3 lockfiles); no other dependencies were moved.

## Testing Performed

- [ ] Terraform plan/apply
- [ ] Blueprint deployment test
- [ ] Unit tests
- [ ] Integration tests
- [ ] Bug fix includes regression test (see [Test
Policy](docs/contributing/testing-validation.md))
- [x] Manual validation
- [ ] Other:

Ran `cargo audit --deny warnings` (with the repo `.github/audit.toml`
allow-list applied) against each affected crate; all three now report no
advisories. The remaining allow-listed advisories (`instant`, `paste`,
`rustls-pemfile`, `rand`) are unchanged.

## Validation Steps

1. For each affected crate, copy the allow-list: `cp .github/audit.toml
<crate>/.cargo/audit.toml`.
2. Run `cargo audit --deny warnings` in:
   - `src/500-application/502-rust-http-connector/services/broker`
   - `src/500-application/507-ai-inference/services/ai-edge-inference`
-
`src/500-application/507-ai-inference/services/ai-edge-inference-crate`
3. Confirm each reports no `error:` advisories.

## Checklist

- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] All new and existing tests passed
- [ ] I have run `terraform fmt` on all Terraform code
- [ ] I have run `terraform validate` on all Terraform code
- [ ] I have run `az bicep format` on all Bicep code
- [ ] I have run `az bicep build` to validate all Bicep code
- [x] I have checked for any sensitive data/tokens that should not be
committed
- [x] Lint checks pass (run applicable linters for changed file types)

## Security Review

- [x] No credentials, secrets, or tokens are hardcoded or logged
- [ ] RBAC and identity changes follow least-privilege principles
- [x] No new network exposure or public endpoints introduced without
justification
- [x] Dependency additions or updates have been reviewed for known
vulnerabilities
- [ ] Container image changes use pinned digests or SHA references

## Additional Notes

- Terraform/Bicep checklist items are not applicable; this change
touches only Rust `Cargo.lock` files.
- This change does not touch any security-sensitive paths
(`SECURITY.md`, `src/000-cloud/010-security-identity/`, `deploy/`), so
the `security-reviewed` label should not be required.
bindsi added 2 commits June 23, 2026 12:21
Add missing transitive dependencies left out of the lock file after
merging main, so cargo llvm-cov --locked no longer fails in the
Rust Tests coverage job.

@bindsi bindsi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — all previously blocking feedback has been addressed and CI is green.

Verified on the current head (e2b52a0):

  • Lockfiles in sync (was blocking): Cargo.lock files were regenerated and all Rust Tests / coverage jobs now pass with cargo llvm-cov --locked. The original --locked failure is resolved.
  • reqwest no longer downgraded: 502-rust-http-connector/services/broker/Cargo.toml now pins reqwest = "0.12.28" (kept on 0.12, json feature added) instead of the earlier 0.11.6. No hyper 0.14/1.x regression.
  • ort default-features concern: the 507-ai-inference/.../ai-edge-inference-crate/Cargo.toml change is no longer in the diff, so load-dynamic is not being silently dropped.
  • Workflow precision fixes applied: the aio-sdks registry grep now excludes commented lines (grep -v '^\s*#') and the Codecov upload is guarded by hashFiles() so skipped crates don't produce noisy failures.

All 5 review threads are resolved and all 61 checks are passing (success/skipped/neutral, none failing or pending). AIO SDK 0.x→1.x upgrades across the .NET and Rust crates look consistent, with matching lockfile updates.

Non-blocking observation for a follow-up: .cargo/config.toml removes the aio-sdks registry definition (packages now resolved from crates.io), while rust-tests.yml still contains the cargo login --registry aio-sdks step and the registry = "aio-sdks" skip guard. These are now effectively dead paths since no Cargo.toml references that registry — safe to leave, but worth pruning later to avoid confusion.

@katriendg katriendg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the iterative work on this 🙏

Big thanks to @kgmwang1 and Marcel Bindseil for pushing this AIO SDK GA migration through a long series of iterations and for cleaning up the merge-conflict artifacts that were breaking CI. The app coverage is spot on — I verified against main that exactly the six src/500-application services that depend on the AIO SDKs (500, 501, 502, 503, 504, 507) are updated, with no app missed, and the 507 mqtt.rs API migration lines up cleanly with the GA SDK surface. 👏

I'm marking this Request changes for a few valid items that should be applied before we merge. They're small and mostly mechanical — I've left inline suggestions where possible.

🔴 Please address before merge

  1. Pin to GA, not betaazure_iot_operations_services is set to 1.3.0-beta1 in 501 (sender + receiver) and 507. crates.io confirms 1.2.0 is the max stable release, which is the GA target in #298. Inline suggestion blocks are attached on all three manifests. If any code path actually requires a 1.3.0-beta1-only API, let's call that out explicitly rather than ship a pre-release.
  2. Remove out-of-scope filessrc/500-application/514-wasm-msg-to-dss/operators/msg-to-dss-key/src/enricher.rs and .../resources/graphs/lowcode-pallet-correlation-enrichment.yaml are new feature files unrelated to the SDK upgrade (#298 explicitly lists new AIO usage as out of scope). Inline comments attached.
  3. Drop the dead aio-sdks CI logic — since every 500-application crate moved off the private registry to crates.io, the CARGO_REGISTRIES_AIO_SDKS_TOKEN secret, the auth step, and the coverage-skip branch in rust-tests.yml are no longer needed. Inline comment attached.

🟠 PR hygiene (required, no inline anchor)

  1. Use the PR template and link the issue — please rewrite the description with the repo PR template and add Closes #298 so it auto-closes on merge.
  2. Capture upgrade notes#298's acceptance criteria asks for notes on non-obvious breaking changes. Worth summarizing the 507/504 API migration (SessionConnectionMonitorSessionMonitor, publishpublish_qos1, new TopicName/TopicFilter/SubscribeProperties/PublishProperties, message.topicmessage.topic_name) and noting that 507 dropped its direct azure_iot_operations_protocol dependency.

✅ Looks good / no action

  • 507 azure_iot_operations_protocol removal is safe — verified no .rs file references it.
  • mqtt (1.0.2) and protocol (1.0.0) pins are GA.
  • Keeping .devcontainer/devcontainer-lock.json is fine.

One thing to keep an eye on: the Dependency Review check still flags 1 vulnerable package, and a RUSTSEC-2026-0186 (memmap2) ignore was added to 507's audit list — acceptable given the documented candle-core constraint, but please acknowledge it in the PR so it's an explicit decision.

Thanks again — once the above are in, this should be ready to go. 🚀

Comment thread src/500-application/507-ai-inference/services/ai-edge-inference/Cargo.toml Outdated
Comment thread src/500-application/501-rust-telemetry/services/sender/Cargo.toml Outdated
Comment thread src/500-application/501-rust-telemetry/services/receiver/Cargo.toml Outdated
Comment thread src/500-application/514-wasm-msg-to-dss/operators/msg-to-dss-key/src/enricher.rs Outdated
Comment thread .github/workflows/rust-tests.yml Outdated
kgmwang1 and others added 7 commits June 24, 2026 03:19
🧹 - Generated by Copilot
downgrade rust telemetry sender and receiver from 1.3.0-beta1 to 1.2.0
note that msg-to-dss enrichment files were removed from this PR scope

Removed from this PR:

src/500-application/514-wasm-msg-to-dss/resources/graphs/lowcode-pallet-correlation-enrichment.yaml
enricher.rs

🔧 - Generated by Copilot
…e/Cargo.toml

Co-authored-by: Katrien De Graeve <katriendg@users.noreply.github.com>
🔧 - Generated by Copilot

Signed-off-by: Marcel Bindseil <marcelbindseil@gmail.com>
@kgmwang1 kgmwang1 requested a review from katriendg June 24, 2026 15:45

@katriendg katriendg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround 🙏

Really nice iteration here, @kgmwang1 (and Marcel Bindseil) — almost everything from the last pass is addressed:

  • azure_iot_operations_services now pinned to GA 1.2.0 across 501 sender/receiver and 507 (and the C# 500-basic-inference is on GA too). No more 1.3.0-beta1.
  • ✅ Both out-of-scope 514-wasm-msg-to-dss files removed.
  • ✅ PR description rewritten with the template, Closes #298, the documented API breaking changes, and the 507 azure_iot_operations_protocol removal note.

One remaining item (then this is good to go)

The rust-tests.yml aio-sdks token/auth handling is still in place, and I'm now confident it should be removed. I validated the registry directly and it requires no auth (config.json returns "auth-required":false; the wasm_graph_sdk index and tarball both download anonymously with 200). The WASM crates that still use registry = "aio-sdks" build fine without a token, and this coverage matrix doesn't even run those crates — it's only 501507 + 901, which all moved to crates.io in this PR. Details in the inline comment.

Once the env, the "Configure aio-sdks registry authentication" step, and the token-based coverage-skip branch are removed (keep --locked), I'm happy to approve. Thanks again! 🚀

Comment thread .github/workflows/rust-tests.yml Outdated
@kgmwang1 kgmwang1 requested a review from katriendg June 25, 2026 16:07

@katriendg katriendg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the additional changes! All good

@kgmwang1 kgmwang1 merged commit 39acee3 into main Jun 25, 2026
61 checks passed
@kgmwang1 kgmwang1 deleted the feat/298-aio-sdk-telemetry-upgrade branch June 25, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore: migrate to GA Azure IoT Operations SDK updates across affected applications in src/500-application

5 participants