Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to build tests for libs on Riscv64 #97791

Open
ashaurtaev opened this issue Jan 31, 2024 · 15 comments
Open

Failed to build tests for libs on Riscv64 #97791

ashaurtaev opened this issue Jan 31, 2024 · 15 comments
Labels
arch-riscv Related to the RISC-V architecture area-Infrastructure
Milestone

Comments

@ashaurtaev
Copy link
Contributor

ashaurtaev commented Jan 31, 2024

After #95980 we get next output during build of CoreCLR

  Determining projects to restore...
  Tool 'coverlet.console' (version '6.0.0') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '5.0.2') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '9.0.0-prerelease.24077.1') was restored. Available commands: xharness
  Tool 'microsoft.visualstudio.slngen.tool' (version '11.1.0') was restored. Available commands: slngen
  
  Restore was successful.
  All projects are up-to-date for restore.
  Determining projects to restore...
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
  Nothing to do. None of the projects specified contain packages to restore.

Build FAILED.

Build commands:

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false

After the first command the build completes successfully.
The error occurs in the following command:

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

Please help to fix it.

cc @gbalykov

@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 31, 2024
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jan 31, 2024
@filipnavara filipnavara added arch-riscv Related to the RISC-V architecture area-Infrastructure and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jan 31, 2024
@ghost
Copy link

ghost commented Jan 31, 2024

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

After #95980 we get next output during build of CoreCLR

  Determining projects to restore...
  Tool 'coverlet.console' (version '6.0.0') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '5.0.2') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '9.0.0-prerelease.24077.1') was restored. Available commands: xharness
  Tool 'microsoft.visualstudio.slngen.tool' (version '11.1.0') was restored. Available commands: slngen
  
  Restore was successful.
  All projects are up-to-date for restore.
  Determining projects to restore...
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
  Nothing to do. None of the projects specified contain packages to restore.

Build FAILED.

Build commands:

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false

After the first command the build completes successfully.
The error occurs in the following command:

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

Please help to fix it.

cc @gbalykov

Author: ashaurtaev
Assignees: -
Labels:

area-Infrastructure, untriaged, arch-riscv

Milestone: -

@ashaurtaev
Copy link
Contributor Author

@ViktorHofer please help, what should I do?

@ViktorHofer
Copy link
Member

@dotnet/runtime-infrastructure can you please help? Seems to be host and prebuilt related. Possibly a component that tries to use a live built host?

@tomeksowi
Copy link
Contributor

tomeksowi commented Feb 5, 2024

Possibly a component that tries to use a live built host?

The one that I know of is src/libraries/System.Runtime.InteropServices/tests/TestAssets/NativeExports/NativeExports.csproj, at least that's what the /p:UseLocalAppHostPack=true was meant to override; not doing so resulted in ComInterfaceGenerator.Tests and LibraryImportGenerator.Tests try to use .so's for x64.

But I don't see it anywhere in build log with --verbosity diag and it also fails without /p:UseLocalAppHostPack=true, so not sure if it's related to this failure.

@ashaurtaev
Copy link
Contributor Author

If option /p:UseLocalAppHostPack=true is not added, then the error still occurs

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false

  Determining projects to restore...
  Tool 'coverlet.console' (version '6.0.0') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '5.0.2') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '9.0.0-prerelease.24077.1') was restored. Available commands: xharness
  Tool 'microsoft.visualstudio.slngen.tool' (version '11.1.0') was restored. Available commands: slngen
  
  Restore was successful.
  All projects are up-to-date for restore.
  Determining projects to restore...
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
  Nothing to do. None of the projects specified contain packages to restore.

Build FAILED.

@ashaurtaev ashaurtaev changed the title RiscV64 build fail Failed to build tests for libs on Riscv64 Feb 8, 2024
@ashaurtaev
Copy link
Contributor Author

@dotnet/area-infrastructure-libraries Could you take a look, please?

@am11
Copy link
Member

am11 commented Mar 4, 2024

@ViktorHofer, this is an variant of #58109. The build flow is twisted, and we need to build it like this:

# using mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-riscv64

$ export ROOTFS_DIR=/crossrootfs/riscv64

# clr cross tools depend on live apphsot and host managed depends on something from sfxproj..
# so we build host.native first
$ ./build.sh host.native --cross --arch riscv64 -c Release

# now build clr+libs+tests
$ ./build.sh clr+libs --cross --arch riscv64 --runtimeConfiguration Release \
    --librariesConfiguration Release --hostConfiguration Release \
    --subset libs.pretest+libs.tests --testscope all -p:UseLocalAppHostPack=true

# now build packs 
$ ./build.sh packs --cross --arch riscv64 -c Release

# finally the host
$ ./build.sh host --cross --arch riscv64 -c Release

I think linux-riscv64 at this point (while it's still a community-supported platform) provides a good opportunity to streamline the subset dependencies, because on other archs apphost gets pulled from SDK (accidentally?) in places where we don't expect.

@clamp03
Copy link
Member

clamp03 commented Mar 5, 2024

cc @dotnet/samsung

@gbalykov
Copy link
Member

gbalykov commented Mar 5, 2024

Original issue with libs tests build was fixed at some point (not sure about exact commit), but before that similar problem with coreclr tests started to happen as I mentioned in #96941 (comment).

Now clr, libs and libs tests build works with next commands without any issues:

ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false

ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

However, trying to build clr tests with next command fails:

ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1

error (same as @clamp03 mentioned in #98476 (comment)):

    Determining projects to restore...
/home/runtime/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj : error NU1101: Unable to find package Microsoft.NETCore.App.Runtime.linux-riscv64. No packages exist with this id in source(s): /home/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/FSharp/library-packs, dotnet-eng, dotnet-libraries, dotnet-libraries-transport, dotnet-public, dotnet-tools, dotnet9, dotnet9-transport [/home/runtime/src/tests/build.proj]
/home/runtime/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-riscv64. No packages exist with this id in source(s): /home/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/FSharp/library-packs, dotnet-eng, dotnet-libraries, dotnet-libraries-transport, dotnet-public, dotnet-tools, dotnet9, dotnet9-transport [/home/runtime/src/tests/build.proj]
    Failed to restore /home/runtime/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 19.4 sec).
/home/runtime/src/tests/build.proj(548,5): error MSB3073: The command ""/home/runtime/.dotnet/dotnet" restore Common/test_dependencies_fs/test_dependencies.fsproj  /p:SetTFMForRestore=true /p:TargetOS=linux /p:TargetArchitecture=riscv64 /p:Configuration=Release /p:CrossBuild=true" exited with code 1.

@am11 do you know what can help with this one?

@ViktorHofer
Copy link
Member

@am11 do you know what can help with this one?

@jkoritzinsky @hoyosjs @trylek might be able to help

@am11
Copy link
Member

am11 commented Mar 5, 2024

@gbalykov, please try applying:

--- a/src/tests/build.proj
+++ b/src/tests/build.proj
@@ -543,6 +543,7 @@
   <Target Name="RestorePackage">
     <PropertyGroup>
       <_ConfigurationProperties>/p:TargetOS=$(TargetOS) /p:TargetArchitecture=$(TargetArchitecture) /p:Configuration=$(Configuration) /p:CrossBuild=$(CrossBuild)</_ConfigurationProperties>
+      <_ConfigurationProperties Condition="'$(UseLocalAppHostPack)' == 'true'">$(_ConfigurationProperties) -p:EnableAppHostPackDownload=false -p:EnableTargetingPackDownload=false -p:EnableRuntimePackDownload=false</_ConfigurationProperties>
       <DotnetRestoreCommand>"$(DotNetTool)" restore $(RestoreProj) $(PackageVersionArg) /p:SetTFMForRestore=true $(_ConfigurationProperties)</DotnetRestoreCommand>
     </PropertyGroup>
     <Exec Command="$(DotnetRestoreCommand)"/>

and add -p:UseLocalAppHostPack=true at the end of your call:

ROOTFS_DIR=pwd/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1

@gbalykov
Copy link
Member

gbalykov commented Mar 6, 2024

@am11 Thanks for quick response, your fix did the trick! Now clr and libs tests can be built for riscv64 without any issues (I checked on f84d33c). Will you open PR for your patch?

For reference, he's a set of commands to build coreclr, libs, coreclr tests and libs tests for riscv64:

# Build coreclr + libs
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false

# Build libs tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

# Build coreclr tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1 -p:UseLocalAppHostPack=true

@ViktorHofer
Copy link
Member

I wonder if this simpler command also works. Would you mind trying?

# Build coreclr + libs
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --configuration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs /p:EnableSourceLink=false

# Build libs tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --configuration Release --subset libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

# Build coreclr tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1 -p:UseLocalAppHostPack=true

@am11
Copy link
Member

am11 commented Mar 6, 2024

Great! Yes, I will send a PR shortly. This seems like a stopgap solution.
I think the proper solution would be to integrate eng/targetingpacks.targets in src/tests and adjust the subset dependencies, so we can utilize apphost live built in various subsets without going through the sdk overhead. Also that "$(DotNetTool)" restore part can probably be replaced by <MSBuild Targets="Restore" to avoid shelling out the restore task.

@gbalykov
Copy link
Member

gbalykov commented Mar 6, 2024

I wonder if this simpler command also works. Would you mind trying?

This also works

@agocke agocke added this to the Future milestone Jul 8, 2024
@agocke agocke removed the untriaged New issue has not been triaged by the area owner label Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-riscv Related to the RISC-V architecture area-Infrastructure
Projects
Status: No status
Development

No branches or pull requests

8 participants