Skip to content

Commit

Permalink
Upgrade to .NET 8 + latest LTS of IoT SDKs (#7262)
Browse files Browse the repository at this point in the history
In preparation for v1.5, this change updates our C# code to .NET 8.0. It also updates our dependency on the IoT SDKs to the latest LTS versions (Microsoft.Azure.Devices.Client 1.36.10, Microsoft.Azure.Devices 1.31.6).

I had to upgrade our dependency on Azure.Storage.Blobs because the device SDK upgraded theirs. This required that I upgrade the code in Edge Agent and some test modules to use newer storage blob libraries that aren't deprecated. I did local testing of the Edge Agent code to ensure that the new calls work as expected.

I updated the buildBranch.sh and runTests.sh scripts. buildBranch.sh was building two different ways: (1) it was doing a standard build of all .NET components from the repo root (`dotnet build -c Release`), and (2) it was publishing each "app" in a manner suitable for including in Docker images. I simplified it to just focus on the second scenario. For the first, I updated dotnet.yaml to call `dotnet test` directly. I simplified runTests.sh to use `dotnet test` instead of `dotnet vstest` (which is being deprecated) because `dotnet test` now supports running pre-built binaries instead of always building the binaries itself. Note that `dotnet test` apparently doesn't support some of the parallelization features that vstest did, but the time difference is minimal and the script logic is vastly simplified.

I updated EdgeHubTriggerCsharp.csproj to specify its own TargetFramework property, rather than including netcoreappVersion.props, because it has dependedencies that don't yet support .NET 8.

I successfully ran the CI Build pipeline with these changes. To test, I ran the end-to-end tests, nested end-to-end tests, connectivity tests, and ISA-95 smoke tests against the new bits.

## Azure IoT Edge PR checklist:
  • Loading branch information
damonbarry committed Apr 12, 2024
1 parent 54e46fd commit 7508ffc
Show file tree
Hide file tree
Showing 76 changed files with 182 additions and 379 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -332,3 +332,6 @@ parts/
stage/
prime/
*.snap

# used by the C# Dev Kit (VS Code extension)
.mono/
23 changes: 7 additions & 16 deletions builds/checkin/dotnet.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,12 @@ jobs:
steps:
- script: scripts/linux/installPrereqs.sh
displayName: Install Prerequisites
- script: scripts/linux/buildBranch.sh -c '$(configuration)'
displayName: Build
- script: scripts/linux/runTests.sh '--filter Category=Unit&Category!=GetLogsTests'
displayName: Test
- script: scripts/linux/runTests.sh '--filter Category=GetLogsTests'
displayName: Run GetLogsTests Tests
- script: |
dotnet test \
--configuration $(configuration) \
--filter 'Category=Unit' \
--logger trx \
displayName: Build and run unit tests
- task: PublishTestResults@2
displayName: Publish Test Results
inputs:
Expand All @@ -69,19 +69,10 @@ jobs:
dotnet test \
--logger trx \
--results-directory '$(Agent.TempDirectory)' \
--filter 'Category=Unit&Category!=GetLogsTests' \
--filter 'Category=Unit' \
--collect 'xplat code coverage' \
--settings CodeCoverage.runsettings
displayName: Run unit tests with code coverage
- script: |
dotnet test 'edge-agent/test/Microsoft.Azure.Devices.Edge.Agent.Core.Test' \
--logger trx \
--results-directory '$(Agent.TempDirectory)' \
--filter 'Category=GetLogsTests' \
--collect 'xplat code coverage' \
--settings CodeCoverage.runsettings
displayName: Run GetLogsTests unit tests with code coverage
condition: succeededOrFailed()
- script: |
dotnet new tool-manifest
dotnet tool install dotnet-reportgenerator-globaltool
Expand Down
24 changes: 1 addition & 23 deletions builds/ci/dotnet.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
CERT_VALUE: $(IotHubMqttHeadCert)
- script: scripts/linux/buildBranch.sh -c $(Build.Configuration) --no-rocksdb-bin --skip-quickstart
displayName: Build
- script: sudo -E bash -c './scripts/linux/runTests.sh "--filter $(test.filter)&Category!=Kubernetes" $(Build.Configuration)'
- script: sudo -E bash -c './scripts/linux/runTests.sh "$(test.filter)" "$(Build.Configuration)"'
displayName: Integration Tests
env:
DummySecret1: $(DummySecret1)
Expand All @@ -59,28 +59,6 @@ jobs:
IotDevice3ConnStr2: $(IotDevice3ConnStr2)
IotHubConnStr2: $(IotHubConnStr2)
IotHubMqttHeadCert: $(IotHubMqttHeadCert)
# - script: |
# # download kind binary and make it executable
# curl -Lo ./kind https://github.com/kubernetes-sigs/kind/releases/download/v0.5.1/kind-$(uname)-amd64
# chmod +x ./kind
# # move it to $AGENT_TOOLSDIRECTORY\kind
# mkdir $AGENT_TOOLSDIRECTORY/kind
# mv ./kind $AGENT_TOOLSDIRECTORY/kind/kind
# # make kind to be available for integration tests
# echo "##vso[task.setvariable variable=PATH;]$AGENT_TOOLSDIRECTORY/kind:$PATH"
# displayName: Install Kubernetes cluster manager (kind)
# condition: succeededOrFailed()
# - script: |
# kind create cluster --name "edgy" --wait 5m
# KUBECONFIG=$(kind get kubeconfig-path --name "edgy")
# echo "##vso[task.setvariable variable=KUBECONFIG;]$KUBECONFIG"
# echo "##vso[task.setvariable variable=USE_EXISTING_KUBERNETES_CLUSTER;]true"
# displayName: Create Kubernetes cluster (edgy)
# condition: succeededOrFailed()
# - script: |
# sudo -E bash -c './scripts/linux/runTests.sh "--filter $(test.filter)&Category=Kubernetes" $(Build.Configuration)'
# displayName: Kubernetes Integration Tests
# condition: succeededOrFailed()
- task: PublishTestResults@2
displayName: Publish test results
inputs:
Expand Down
2 changes: 1 addition & 1 deletion builds/e2e/templates/e2e-setup.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ steps:
$testDir = '$(Build.SourcesDirectory)/test/Microsoft.Azure.Devices.Edge.Test'
dotnet build $testDir
$binDir = Convert-Path "$testDir/bin/Debug/net6.0"
$binDir = Convert-Path "$testDir/bin/Debug/net8.0"
Write-Output "##vso[task.setvariable variable=binDir]$binDir"
env:
http_proxy: $(Agent.ProxyUrl)
Expand Down
2 changes: 1 addition & 1 deletion doc/devguide.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ scripts/linux/downloadAndInstallCert.sh -v <VaultName> -c <CertName>
Then run the tests either with Test Explorer in Visual Studio IDE, or with:

```sh
scripts/linux/runTests.sh "--filter Category=Integration"
scripts/linux/runTests.sh "Category=Integration"
```

The syntax of the "filter" argument is described [here](https://docs.microsoft.com/dotnet/core/tools/dotnet-test#filter-option-details). All IoT Edge tests are categorized as one of `Unit`, `Integration`, or `Bvt`.
Expand Down
2 changes: 1 addition & 1 deletion edge-agent/docker/linux/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# syntax=docker/dockerfile:1.4

FROM mcr.microsoft.com/dotnet/aspnet:6.0-alpine
FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine

ARG EXE_DIR=.
ARG TARGETPLATFORM
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.36.8" />
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.36.10" />
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="5.0.0" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.2" />
<PackageReference Include="System.Linq.Async" Version="4.1.1" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,23 @@
namespace Microsoft.Azure.Devices.Edge.Agent.IoTHub
{
using System;
using System.IO;
using System.Threading.Tasks;
using global::Azure.Storage.Blobs.Specialized;
using Microsoft.Azure.Devices.Edge.Util;
using Microsoft.WindowsAzure.Storage.Blob;

class AzureAppendBlob : IAzureAppendBlob
{
readonly CloudAppendBlob appendBlob;
readonly AppendBlobClient appendBlob;

public AzureAppendBlob(CloudAppendBlob appendBlob)
public AzureAppendBlob(AppendBlobClient appendBlob)
{
this.appendBlob = Preconditions.CheckNotNull(appendBlob, nameof(appendBlob));
}

public string Name => this.appendBlob.Name;

public Task AppendByteArray(ArraySegment<byte> bytes) => this.appendBlob.AppendFromByteArrayAsync(bytes.Array, bytes.Offset, bytes.Count);
public Task AppendByteArray(ArraySegment<byte> bytes) =>
this.appendBlob.AppendBlockAsync(new MemoryStream(bytes.Array, bytes.Offset, bytes.Count));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.5" />
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.36.8" />
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.6" />
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.36.10" />
<PackageReference Include="Azure.Storage.Blobs" Version="12.19.1" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,29 @@ namespace Microsoft.Azure.Devices.Edge.Agent.IoTHub.Blob
{
using System.IO;
using System.Threading.Tasks;
using global::Azure.Storage.Blobs.Models;
using global::Azure.Storage.Blobs.Specialized;
using Microsoft.Azure.Devices.Edge.Util;
using Microsoft.WindowsAzure.Storage.Blob;

class AzureBlob : IAzureBlob
{
readonly CloudBlockBlob blockBlob;
readonly BlockBlobClient blockBlob;
readonly BlobHttpHeaders headers;

public AzureBlob(CloudBlockBlob blockBlob)
public AzureBlob(BlockBlobClient blockBlob, BlobHttpHeaders headers)
{
this.blockBlob = Preconditions.CheckNotNull(blockBlob, nameof(blockBlob));
this.headers = Preconditions.CheckNotNull(headers, nameof(headers));
}

public string Name => this.blockBlob.Name;

public Task UploadFromByteArrayAsync(byte[] bytes) => this.blockBlob.UploadFromByteArrayAsync(bytes, 0, bytes.Length);
public Task UploadFromByteArrayAsync(byte[] bytes)
{
var options = new BlobUploadOptions { HttpHeaders = this.headers };
return this.blockBlob.UploadAsync(new MemoryStream(bytes), options);
}

public Task UploadFromStreamAsync(Stream source) => this.blockBlob.UploadFromStreamAsync(source);
public Task UploadFromStreamAsync(Stream source) => this.blockBlob.UploadAsync(source);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ namespace Microsoft.Azure.Devices.Edge.Agent.IoTHub.Blob
using System.Globalization;
using System.IO;
using System.Threading.Tasks;
using global::Azure.Storage.Blobs;
using Microsoft.Azure.Devices.Edge.Agent.Core;
using Microsoft.Azure.Devices.Edge.Agent.Core.Logs;
using Microsoft.Azure.Devices.Edge.Util;
using Microsoft.Azure.Devices.Edge.Util.TransientFaultHandling;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage.Blob;

public class AzureBlobRequestsUploader : IRequestsUploader
{
Expand Down Expand Up @@ -47,7 +47,7 @@ public async Task UploadLogs(string uri, string id, byte[] payload, LogsContentE
{
var containerUri = new Uri(uri);
string blobName = this.GetBlobName(id, GetLogsExtension(logsContentEncoding, logsContentType));
var container = new CloudBlobContainer(containerUri);
var container = new BlobContainerClient(containerUri);
Events.Uploading(blobName, container.Name);

await ExecuteWithRetry(
Expand All @@ -74,7 +74,7 @@ public async Task UploadLogs(string uri, string id, byte[] payload, LogsContentE

var containerUri = new Uri(uri);
string blobName = this.GetBlobName(id, GetLogsExtension(logsContentEncoding, logsContentType));
var container = new CloudBlobContainer(containerUri);
var container = new BlobContainerClient(containerUri);
Events.Uploading(blobName, container.Name);
IAzureAppendBlob blob = await this.azureBlobUploader.GetAppendBlob(containerUri, blobName, GetContentType(logsContentType), GetContentEncoding(logsContentEncoding));
return blob.AppendByteArray;
Expand All @@ -89,7 +89,7 @@ public async Task UploadSupportBundle(string uri, Stream source)
{
var containerUri = new Uri(uri);
string blobName = this.GetBlobName("support-bundle", "zip");
var container = new CloudBlobContainer(containerUri);
var container = new BlobContainerClient(containerUri);
Events.Uploading(blobName, container.Name);
await ExecuteWithRetry(
() =>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,33 @@ namespace Microsoft.Azure.Devices.Edge.Agent.IoTHub.Blob
{
using System;
using System.Threading.Tasks;
using global::Azure.Storage.Blobs;
using global::Azure.Storage.Blobs.Models;
using global::Azure.Storage.Blobs.Specialized;
using Microsoft.Azure.Devices.Edge.Util;
using Microsoft.WindowsAzure.Storage.Blob;

class AzureBlobUploader : IAzureBlobUploader
{
public IAzureBlob GetBlob(Uri containerUri, string blobName, Option<string> contentType, Option<string> contentEncoding)
{
var container = new CloudBlobContainer(Preconditions.CheckNotNull(containerUri, nameof(containerUri)));
CloudBlockBlob blob = container.GetBlockBlobReference(Preconditions.CheckNonWhiteSpace(blobName, nameof(blobName)));
contentType.ForEach(c => blob.Properties.ContentType = c);
contentEncoding.ForEach(c => blob.Properties.ContentEncoding = c);
var azureBlob = new AzureBlob(blob);
var container = new BlobContainerClient(Preconditions.CheckNotNull(containerUri, nameof(containerUri)));
BlockBlobClient blob = container.GetBlockBlobClient(Preconditions.CheckNonWhiteSpace(blobName, nameof(blobName)));
var headers = new BlobHttpHeaders();
contentType.ForEach(c => headers.ContentType = c);
contentEncoding.ForEach(c => headers.ContentEncoding = c);
var azureBlob = new AzureBlob(blob, headers);
return azureBlob;
}

public async Task<IAzureAppendBlob> GetAppendBlob(Uri containerUri, string blobName, Option<string> contentType, Option<string> contentEncoding)
{
var container = new CloudBlobContainer(Preconditions.CheckNotNull(containerUri, nameof(containerUri)));
CloudAppendBlob blob = container.GetAppendBlobReference(Preconditions.CheckNonWhiteSpace(blobName, nameof(blobName)));
contentType.ForEach(c => blob.Properties.ContentType = c);
contentEncoding.ForEach(c => blob.Properties.ContentEncoding = c);
await blob.CreateOrReplaceAsync();
var container = new BlobContainerClient(Preconditions.CheckNotNull(containerUri, nameof(containerUri)));
AppendBlobClient blob = container.GetAppendBlobClient(Preconditions.CheckNonWhiteSpace(blobName, nameof(blobName)));
var headers = new BlobHttpHeaders();
contentType.ForEach(c => headers.ContentType = c);
contentEncoding.ForEach(c => headers.ContentEncoding = c);
var options = new AppendBlobCreateOptions { HttpHeaders = headers };
await blob.CreateAsync(options);
var azureBlob = new AzureAppendBlob(blob);
return azureBlob;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

<ItemGroup>
<PackageReference Include="Microsoft.Azure.Amqp" Version="2.5.12" />
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.5" />
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.6" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.5" />
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.6" />
<PackageReference Include="Stateless" Version="4.0.0" />
</ItemGroup>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,21 +18,8 @@
<ItemGroup>
<PackageReference Include="App.Metrics" Version="3.0.0" />
<PackageReference Include="JetBrains.Annotations" Version="2018.3.0" />
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.36.8" />
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.36.10" />
<PackageReference Include="System.ValueTuple" Version="4.5.0" />
<!--
We have a transitive dependency on Dotnetty via Microsoft.Azure.Devices.Client. Use
explicit references here to override the version and fix a known OOM bug. See
https://github.com/Azure/DotNetty/pull/583. If Microsoft.Azure.Devices.Client updates
its dependency, we can remove the Dotnetty references.
-->
<PackageReference Include="DotNetty.Buffers" Version="0.7.5" />
<PackageReference Include="DotNetty.Codecs" Version="0.7.5" />
<PackageReference Include="DotNetty.Codecs.Mqtt" Version="0.7.5" />
<PackageReference Include="DotNetty.Common" Version="0.7.5" />
<PackageReference Include="DotNetty.Handlers" Version="0.7.5" />
<PackageReference Include="DotNetty.Transport" Version="0.7.5" />

</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
</ItemGroup>

<ItemGroup>
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.5" />
<PackageReference Include="Microsoft.Azure.Devices" Version="1.31.6" />
</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -536,7 +536,7 @@ async Task SendResponseAsync(HttpStatusCode status, Dictionary<string, string> h

foreach (var header in headers)
{
this.Response.Headers.Add(header.Key, header.Value);
this.Response.Headers.Append(header.Key, header.Value);
}

if (!string.IsNullOrEmpty(jsonContent))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,6 @@
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="5.0.0" />
<PackageReference Include="Microsoft.Azure.Devices.ProtocolGateway.Core" Version="2.0.1" />
<!--
We have a transitive dependency on Dotnetty via Microsoft.Azure.Devices.ProtocolGateway.Core.
Use explicit references here to override the version and fix a known OOM bug. See
https://github.com/Azure/DotNetty/pull/583. If ProtocolGateway updates its dependency,
we can remove the Dotnetty references.
-->
<PackageReference Include="DotNetty.Buffers" Version="0.7.5" />
<PackageReference Include="DotNetty.Codecs" Version="0.7.5" />
<PackageReference Include="DotNetty.Codecs.Mqtt" Version="0.7.5" />
<PackageReference Include="DotNetty.Common" Version="0.7.5" />
<PackageReference Include="DotNetty.Handlers" Version="0.7.5" />
<PackageReference Include="DotNetty.Transport" Version="0.7.5" />

</ItemGroup>

<ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@
namespace Microsoft.Azure.Devices.Edge.Hub.Service
{
using System;
using System.Linq;
using System.Threading.Tasks;
using Autofac;
using Autofac.Extensions.DependencyInjection;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Devices.Edge.Hub.Core;
using Microsoft.Azure.Devices.Edge.Hub.Http;
Expand Down Expand Up @@ -86,7 +88,7 @@ public void Configure(IApplicationBuilder app)
async (context, next) =>
{
// Response header is added to prevent MIME type sniffing
context.Response.Headers.Add("X-Content-Type-Options", "nosniff");
context.Response.Headers.Append("X-Content-Type-Options", "nosniff");
await next();
});

Expand Down

0 comments on commit 7508ffc

Please sign in to comment.