Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bootstrapped smoke-test running unable to find package Microsoft.DotNet.Arcade.Sdk #2958

Closed
mirespace opened this issue Aug 2, 2022 · 6 comments
Assignees
Labels
area-testing Improvements in CI and testing

Comments

@mirespace
Copy link

Hi team!

When running smoke test for v6.0.107 on Ubuntu with the following command using the tarball created from monorepo:

VERBOSE=1 ./build.sh --run-smoke-test --with-sdk previously-built-dotnet --with-packages packages/archive/

where previously-built-dotnet is a copy of the dotnet /usr/lib/dotnet/dotnet6-6.0.107 folder and packages archive has both source artifacts prebuilts compressed and uncomppresed (I mean the packages/archive/Private.SourceBuilt.Artifacts.6.0.107.tar.gz and its contents uncompressed)

I got the following error:
/root/SRU-testing/dotnet6-6.0.107/Directory.Build.props(9,3): error : Unable to find package Microsoft.DotNet.Arcade.Sdk. No packages exist with this id in source(s): nuget.org

Looking for the error I found dotnet/aspnetcore#11802 and dotnet/roslyn#38884 , but I checked the Nuget.Config files inside src/arcade folder just in case and I don't find any overrides or disabled packages downloads.

The log file for the test is the following:

Build started 08/02/2022 16:37:56.
Logging verbosity is set to: Detailed.Process = "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/dotnet"
MSBuild executable path = "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/sdk/6.0.107/MSBuild.dll"
Command line arguments = "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/sdk/6.0.107/MSBuild.dll /root/SRU-testing/dotnet6-6.0.107/build.proj /bl:/root/SRU-testing/dotnet6-6.0.107/artifacts/log/Debug/BuildTests_0802163756.binlog /fileLoggerParameters:LogFile=/root/SRU-testing/dotnet6-6.0.107/artifacts/logs/BuildTests_0802163756.log /clp:v=m /flp:v=detailed /t:RunSmokeTest /p:CustomPrebuiltSourceBuiltPackagesPath=/root/SRU-testing/dotnet6-6.0.107/packages/archive"
Current directory = "/root/SRU-testing/dotnet6-6.0.107"
MSBuild version = "17.0.0+c9eb9dd64"
Resolving SDK 'Microsoft.NET.Sdk'...
Importing project "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/sdk/6.0.107/Sdks/Microsoft.NET.Sdk/Sdk/Sdk.props" into project "/root/SRU-testing/dotnet6-6.0.107/build.proj" at (2,3).
Project "$(AlternateCommonProps)" was not imported by "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/sdk/6.0.107/Sdks/Microsoft.NET.Sdk/Sdk/Sdk.props" at (39,3), due to false condition; ('$(AlternateCommonProps)' != '') was evaluated as ('' != '').
Importing project "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/sdk/6.0.107/Current/Microsoft.Common.props" into project "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/sdk/6.0.107/Sdks/Microsoft.NET.Sdk/Sdk/Sdk.props" at (40,3).
Importing project "/root/SRU-testing/dotnet6-6.0.107/Directory.Build.props" into project "/root/SRU-testing/dotnet6-6.0.107/previously-built-dotnet/sdk/6.0.107/Current/Microsoft.Common.props" at (33,3).
Resolving SDK 'Microsoft.DotNet.Arcade.Sdk'...
Running restore with 16 concurrent jobs.
Reading project file /tmp/NuGetScratch/c66cabfa0d1d4ae1b0ba4ee392c28d6c/461a0c9949404d22a789e1d05a76a118.proj.
Restoring packages for /tmp/NuGetScratch/c66cabfa0d1d4ae1b0ba4ee392c28d6c/461a0c9949404d22a789e1d05a76a118.proj...
Restoring packages for .NETStandard,Version=v0.0...
  GET https://api.nuget.org/v3-flatcontainer/microsoft.dotnet.arcade.sdk/index.json
  NotFound https://api.nuget.org/v3-flatcontainer/microsoft.dotnet.arcade.sdk/index.json 327ms
Resolving conflicts for .NETStandard,Version=v0.0...
/root/SRU-testing/dotnet6-6.0.107/Directory.Build.props(9,3): error : Unable to find package Microsoft.DotNet.Arcade.Sdk. No packages exist with this id in source(s): nuget.org
Project "/root/SRU-testing/dotnet6-6.0.107/build.proj" on node 1 (RunSmokeTest target(s)).
/root/SRU-testing/dotnet6-6.0.107/Directory.Build.props(9,78): error MSB4236: The SDK 'Microsoft.DotNet.Arcade.Sdk' specified could not be found. [/root/SRU-testing/dotnet6-6.0.107/build.proj]
Done Building Project "/root/SRU-testing/dotnet6-6.0.107/build.proj" (RunSmokeTest target(s)) -- FAILED.
Deferred Messages

Detailed Build Summary
======================


  ============================== Build Hierarchy (IDs represent configurations) =====================================================
  Id                  : Exclusive Time   Total Time   Path (Targets)
  -----------------------------------------------------------------------------------------------------------------------------------
  0                   : 1.093s           1.093s       /root/SRU-testing/dotnet6-6.0.107/build.proj (RunSmokeTest)

  ============================== Node Utilization (IDs represent configurations) ====================================================
  Timestamp:            1        Duration   Cumulative
  -----------------------------------------------------------------------------------------------------------------------------------
  637950550766780904:   0        1.096s     1.096s .....................
  -----------------------------------------------------------------------------------------------------------------------------------
  Utilization:          100.0    Average Utilization: 100.0

Build FAILED.

  /root/SRU-testing/dotnet6-6.0.107/Directory.Build.props(9,3): error : Unable to find package Microsoft.DotNet.Arcade.Sdk. No packages exist with this id in source(s): nuget.org


"/root/SRU-testing/dotnet6-6.0.107/build.proj" (RunSmokeTest target) (1) ->
  /root/SRU-testing/dotnet6-6.0.107/Directory.Build.props(9,78): error MSB4236: The SDK 'Microsoft.DotNet.Arcade.Sdk' specified could not be found. [/root/SRU-testing/dotnet6-6.0.107/build.proj]

    0 Warning(s)
    2 Error(s)

Time Elapsed 00:00:01.13
                               

In the Directory.Build.props mentioned file I see this maybe related to it:

  <Import Condition="'$(SkipArcadeSdkImport)' != 'true'" Project="Sdk.props" Sdk="Microsoft.DotNet.Arcade.Sdk" />
  <PropertyGroup>
    <!-- Fake, for SDK. -->
    <TargetFramework>netstandard2.0</TargetFramework>
    <!-- We have no projects targeting multiple frameworks, so don't include in output path. -->
    <AppendTargetFrameworkToOutputPath>false</AppendTargetFrameworkToOutputPath>
  </PropertyGroup>

Maybe I'm missing any step? Or there is something missing in the bootstrap? Could you please take a look on it andd let me know what you think or what you can think of that might be going on?

Thanks in advance. Have a good day.

@dotnet-issue-labeler dotnet-issue-labeler bot added area-testing Improvements in CI and testing untriaged labels Aug 2, 2022
@crummel
Copy link
Contributor

crummel commented Aug 4, 2022

Hi Miriam, I'm taking a look at this now, thanks for reporting it.

@crummel
Copy link
Contributor

crummel commented Aug 12, 2022

Hi @mirespace , I think I have this figured out. Are you running this is a separate directory from the one you run ./prep.sh --bootstrap in? I think the NuGet.Config file with the feeds for Arcade are not present because it only gets copied over here in the bootstrap-prep case. I think the right fix here is for us to write out a NuGet.Config file and substitute in the directory if you use --with-packages but I think you should be able to try this out using a NuGet.Config file in the directory you're running in an putting in the content:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="previously-source-built" value="<your fully-qualified packages feed here>" />
  </packageSources>
</configuration>

Let me know if you'd like to try that and in the meantime I'll try and get a proper fix ready.

@MichaelSimons
Copy link
Member

@crummel - Can you please backport the fix to 6.0.1xx?

@MichaelSimons
Copy link
Member

This was backported with dotnet/installer#14587 - closing.

@dviererbe
Copy link

Hi 👋🏻

I want to continue where Miriam left off and integrate the smoke-tests into the Ubuntu CI, but as Miriam I have problems running them.

I wrote a Containerfile to make it easier for you to replicate/debug my problem (see below).

When I run

podman run --rm -it --network none dotnet6-smoke-tests

I get the following output:

Using custom bootstrap SDK from '/usr/lib/dotnet', version '6.0.119'
Found bootstrap SDK 6.0.119, bootstrap Arcade 6.0.0-beta.23313.5, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.1+b177f8fa7 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/lib/dotnet/sdk/6.0.119/MSBuild.dll /bl:/tmp/sdk-under-test/source/artifacts/log/Debug/BuildTests_0717064737.binlog /clp:v=m /fileLoggerParameters:LogFile=/tmp/sdk-under-test/source/artifacts/logs/BuildTests_0717064737.log /flp:v=detailed /p:CustomSourceBuiltPackagesPath=/tmp/sdk-under-test/source/packages/archive /t:RunSmokeTest /tmp/sdk-under-test/source/build.proj
/tmp/sdk-under-test/source/Directory.Build.props(9,3): warning : Unable to load the service index for source https://api.nuget.org/v3/index.json.
/tmp/sdk-under-test/source/Directory.Build.props(9,3): error : Unable to find package Microsoft.DotNet.Arcade.Sdk. No packages exist with this id in source(s): nuget.org
/tmp/sdk-under-test/source/Directory.Build.props(9,78): error MSB4236: The SDK 'Microsoft.DotNet.Arcade.Sdk' specified could not be found. [/tmp/sdk-under-test/source/build.proj]

The script tries to download the package Microsoft.DotNet.Arcade.Sdk and not use the offline packages.

I already spend a lot of time set various environment variables, script parameter & file layouts, but I could not get the somke tests running. Could you point me in the right direction or show me a minimal working example?


Note: This Containerimage takes ages to build. You can pull it from updf8ehoay.azurecr.io/dotnet6-smoke-tests to save time.

Note: If you want to build this image yourself, note the COPY dotnet-smoke-test-prereqs.6.0.120.tar.gz .; you have to provide this file yourself.

FROM ubuntu:mantic
ARG SDK_VERSION_UNDER_TEST=6.0.120-0ubuntu1
ARG PREVIOUS_SDK_VERSION=6.0.119-0ubuntu1

# Install dependencies:
RUN apt-get update && apt-get install --assume-yes ubuntu-dev-tools quilt libicu72

# Install previous SDK:
WORKDIR /tmp/previous-sdk
RUN pull-lp-debs dotnet6 $PREVIOUS_SDK_VERSION 
RUN apt install --assume-yes \
    ./dotnet-host*.deb \
    ./dotnet-hostfxr-6.0*.deb \
    ./dotnet-runtime-6.0*.deb \
    ./aspnetcore-runtime-6.0*.deb \
    ./aspnetcore-targeting-pack-6.0*.deb \
    ./dotnet-apphost-pack-6.0*.deb \
    ./dotnet-targeting-pack-6.0*.deb \
    ./netstandard-targeting-pack-2.1*.deb \
    ./dotnet-templates-6.0*.deb \
    ./dotnet-sdk-6.0*.deb

# Download SDK under test source-built artifacts & source code:
WORKDIR /tmp/sdk-under-test
RUN pull-lp-debs dotnet-sdk-6.0-source-built-artifacts $SDK_VERSION_UNDER_TEST
RUN dpkg --install ./dotnet-sdk-6.0-source-built-artifacts*.deb
RUN pull-lp-source dotnet6 $SDK_VERSION_UNDER_TEST
RUN mv dotnet6-* source

# Cleanup unnecessary files
RUN rm /tmp/previous-sdk/*.deb /tmp/sdk-under-test/*.deb

# Link to the source built artifacts and unpack:
WORKDIR /tmp/sdk-under-test/source/packages/archive
RUN ln -s /usr/lib/dotnet/source-built-artifacts/Private.SourceBuilt.Artifacts.$(echo $SDK_VERSION_UNDER_TEST | grep --only-matching --perl-regexp '6.0.1\d\d').tar.gz .
RUN tar --extract --gzip --file Private.SourceBuilt.Artifacts.*.tar.gz

# Import and unpack the smoke test prereqs:
WORKDIR /tmp/sdk-under-test/source/packages/smoke-test-prereqs
COPY dotnet-smoke-test-prereqs.6.0.120.tar.gz .
RUN tar --extract --gzip --file ./dotnet-smoke-test-prereqs.*.tar.gz

# Disable welcome text:
ENV DOTNET_NOLOGO=true
ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true

# Set smoke test environment variables:
ENV VERBOSE=1 
ENV SMOKE_TESTS_PREREQS_PATH=/tmp/sdk-under-test/source/packages/smoke-test-prereqs

# Run Smoke Tests:
WORKDIR /tmp/sdk-under-test/source
ENTRYPOINT [ "/tmp/sdk-under-test/source/build.sh", \
    "--run-smoke-test", \
    "--with-sdk", "/usr/lib/dotnet", \
    "--with-packages", "/tmp/sdk-under-test/source/packages/archive" \
    "--", \
	"/v:n", \
	"/p:SkipPortableRuntimeBuild=true", \
	"/p:LogVerbosity=n", \
	"/p:MinimalConsoleLogOutput=false", \
    "/p:ContinueOnPrebuiltBaselineError=true", \ 
]

@mthalman
Copy link
Member

@dviererbe - Ultimately, the work to allow the smoke tests to be run completely offline is part of #3015.

I had begun making that happen with the changes in dotnet/installer#16359. But that had to be abandoned because the entire approach of maintaining the list of smoke test prereqs was flawed. However, the changes there illustrate some of the work that is necessary for tests to be offline-capable.

I think what you're running into here is that it's implicitly running dotnet restore for the Microsoft.DotNet.SourceBuild.SmokeTests.csproj project. There is no NuGet.config in context, so it automatically generates one that points to nuget.org for the feed. You can see in my changes, that I define a local/offline version of NuGet.config file that points to the prereqs path. That will allow the restore to work offline by pulling the packages from that path. However, the smoke test prereqs tarball only contains the packages required by the tests, not the test project. So you still need to populate the prereqs path with the packages that would be restored as a result of running dotnet restore on the test project. This is what my changes in the prep script did for you. It restored the test project, targeting the smoke test prereqs path as the output location. This ultimately means that you need to do this step in an online environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-testing Improvements in CI and testing
Projects
Archived in project
Development

No branches or pull requests

5 participants