Unable to publish-module to NuGet with a required module dependency

[mmisztal1980]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.

Steps to reproduce

I'm working on emitting powershell modules from a github repo to nuget. My repo contains 2 modules:

• A: CloudTek.Automation.Shell
• B: CloudTek.Automatuon.K8S

B has a dependency on previously published version of A, therefore I've added it to it's RequiredModules . I've encountered a problem during Publish-Module and I'm attempting to solve it. If my understanding is correct, in order to successfully publish B with Publish-Module, A needs to be installed locally 1st as indicated here.

In my CI I'm using PowershellGet 3.0.17-beta17.

Expected behavior

I'm expecting `Install-Module` on A to succeed while installing B locally from a path on the local filesystem of a GitHub actions runner (ubuntu-latest), so that I can execute `Publish-Module` on B in the next step.

Actual behavior

The pipeline fails as seen here

Line |
   2 |  ./Publish.ps1 `
     |  ~~~~~~~~~~~~~~~
     | Failed to publish module CloudTek.Automation.K8S :   The specified
     | RequiredModules entry 'CloudTek.Automation.Shell' in the module manifest
     | '/home/runner/work/automation/automation/src/CloudTek.Automation.K8S/CloudTek.Automation.K8S.psd1' is invalid. Try again after updating this entry with valid values.

Error details

Exception             : System.Exception: No match was found for the specified 
                        search criteria and module name 'CloudTek.Automation.Sh
                        ell'. Try Get-PSRepository to see all available registe
                        red module repositories.
TargetObject          : Microsoft.PowerShell.PackageManagement.Cmdlets.InstallP
                        ackage
CategoryInfo          : ObjectNotFound: (Microsoft.PowerShel…lets.InstallPackag
                        e:InstallPackage) [Install-Package], Exception
FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageMan
                        agement.Cmdlets.InstallPackage
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at Install-Module<Process>, /home/runner/.local/share/p
                        owershell/Modules/PowerShellGet/2.2.5/PSModule.psm1: li
                        ne 9711
                        at <ScriptBlock>, /home/runner/work/automation/automati
                        on/src/CloudTek.Automation.K8S/PrePublish.ps1: line 16
                        at <ScriptBlock>, /home/runner/work/automation/automati
                        on/src/CloudTek.Automation.K8S/PrePublish.ps1: line 7
                        at <ScriptBlock>, /home/runner/work/automation/automati
                        on/scripts/Publish.ps1: line 31
                        at <ScriptBlock>, /home/runner/work/_temp/476525ff-140e
                        -429e-810c-50cafc565387.ps1: line 2
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {0, 1}

Environment data

Key   : PSVersion
Value : 7.2.6
Name  : PSVersion


Key   : PSEdition
Value : Core
Name  : PSEdition


Key   : GitCommitId
Value : 7.2.6
Name  : GitCommitId


Key   : OS
Value : Linux 5.15.0-1022-azure #27~20.04.1-Ubuntu SMP Mon Oct 17 02:03:50 UTC 
        2022
Name  : OS


Key   : Platform
Value : Unix
Name  : Platform


Key   : PSCompatibleVersions
Value : {1.0, 2.0, 3.0, 4.0…}
Name  : PSCompatibleVersions


Key   : PSRemotingProtocolVersion
Value : 2.3
Name  : PSRemotingProtocolVersion


Key   : SerializationVersion
Value : 1.1.0.1
Name  : SerializationVersion


Key   : WSManStackVersion
Value : 3.0
Name  : WSManStackVersion

Visuals

No response

[anamnavi]

@mmisztal1980 I see you've updated to PowerShellGet V3 (with version 3.0.17-beta17), but I should note that PowerShellGetV3 was a rewrite and the cmdlets' names have changed from PowerShellGetV2. So Publish-Module is now Publish-PSResource. So if you can try running Publish-PSResource -Path <path_To_.psd1 or path_to_folder_with_.psd1> and let us know if this still reproduces that would be great, thanks!

Can you also share the module manifest (.psd1) file's contents with us please? Thanks!

[mmisztal1980]

@anamnavi I'm getting a similar result: https://github.com/cloud-tek/automation/actions/runs/3436020892/jobs/5729064700

The manifest can be seen here: https://github.com/cloud-tek/automation/blob/main/src/CloudTek.Automation.K8S/CloudTek.Automation.K8S.psd1

Please note that versioning is achieved using gitversion in CI

[anamnavi]

@mmisztal1980 thanks for sharing this info! We'll work to resolve this soon

[mmisztal1980]

@anamnavi looking forward to hear from you. Meanwhile, can you provide some guidance on how this should be done? I'll try to make sure that my impl is as close to your guidance as possible

[anamnavi]

@mmisztal1980 my apologies for the confusion, specifying ModuleVersion with your RequiredModules should work actually. Here's an example of a package we have that has RequiredModules specified: https://www.powershellgallery.com/packages/PowerShellGet/2.2.5/Content/PowerShellGet.psd1

Could you try using single quotes around the ModuleName and ModuleVersion values. Also could you remove the ; at the end of ModuleVersion (the last key value pair) in your example. So your RequiredModules value should look like this:

  RequiredModules = @(
    @{
      ModuleName = 'CloudTek.Automation.Shell';
      ModuleVersion = '0.1.119'
    }
  )

[mmisztal1980]

@anamnavi I've applied your suggestions. Unfortunately it's failing.

https://github.com/cloud-tek/automation/actions/runs/3527879669/jobs/5917407395

Line |
   2 |  ./Publish.ps1 `
     |  ~~~~~~~~~~~~~~~
     | Failed to publish module CloudTek.Automation.K8S :   The specified
     | RequiredModules entry 'CloudTek.Automation.Shell' in the module manifest
     | '/home/runner/work/automation/automation/src/CloudTek.Automation.K8S/CloudTek.Automation.K8S.psd1' is invalid. Try again after updating this entry with valid values. Run 'Test-ModuleManifest' to validate the module manifest.

[anamnavi]

@mmisztal1980 Your syntax of the .psd1 is now correct and that's not the issue. The issue is that it needs to find the RequiredModule you specify on your system. Looking at your failed job it first fails to install CloudTek.Automation.Shell Version="0.1.119" on L96. That's why it later fails to publish the CloudTek.Automation package which takes a dependency on CloudTek.Automation.Shell...it should be able to install that package from nuget.org.

You could install this with PowerShellGet V3 using the following commands:
Register-PSResourceRepository -Name "NuGetGallery" -Uri "https://api.nuget.org/v3/index.json" -Trusted
Install-PSResource -Name "CloudTek.Automation.Shell" -Version "0.1.119" -Repository NuGetGallery -Verbose

Once you resolve that issue and CloudTek.Automation.Shell is installed, the publish step should work.

I understand the PowerShell message saying the .psd1 entry is invalid is not clear/intuitive enough and should instead indicate that the requiredModule entry package was not installed...we'll fix this aspect.

[mmisztal1980]

@anamnavi let me update the code and try it out really quick

[mmisztal1980]

@anamnavi it appears to have worked https://github.com/cloud-tek/automation/actions/runs/3568486063/jobs/5997407163

I have some questions:

• Can you explain the difference between the following cmdlets?
  • Register-PSRepository
  • Register-PSResourceRepository
• Can Register-PSResourceRepository be used when adding repositories from local filesystem? In a similar way to: Register-PSRepository -Name $_ -SourceLocation "$PSScriptRoot/../$_" ?

[mmisztal1980]

@anamnavi a problem I'm noticing in CI is:

• dependency (A) is pushed to nuget
• install-module (A) is called before the package is available in the nuget feed (my latest package took 10-15mins to become visible

Q:

• How can I ensure that I can install the module from the local filesystem?
• What is the expected folder structure of the ps resource repository folder?

[mmisztal1980]

@anamnavi FYI, I've updated the upstream

• refactoring
• attempts at installing the module locally

[mmisztal1980]

@anamnavi I believe the issue may be hidden in registering the local PSResourceRepository

https://github.com/cloud-tek/automation/actions/runs/3615952444/jobs/6093460572

After registration, Get-PSResourceRepository yields:

Name                            Uri                                                                          Trusted Pri
                                                                                                                     ori
                                                                                                                     ty
----                            ---                                                                          ------- ---
CloudTek.Automation.Shell-local file:///home/runner/work/automation/automation/src/CloudTek.Automation.Shell True    10
PSGallery                       https://www.powershellgallery.com/api/v2                                     True    20
nuget                           https://api.nuget.org/v3/index.json                                          True    40
NuGetGallery                    https://api.nuget.org/v3/index.json                                          True    50

However running Find-PSResource -Name "CloudTek*" against those repositories does not detect the local package in the CloudTek.Automation.Shell-local repository

Name                      Version   Prerelease Repository   Description
----                      -------   ---------- ----------   -----------
CloudTek.Automation.Shell 0.1.152.0            nuget        This package contains reusable powershell automation for ex…
CloudTek.Automation.K8S   0.1.68.0             nuget        This package contains reusable kubernetes powershell automa…
CloudTek.Build            0.11.3.0             nuget        Package Description
CloudTek.Automation.Shell 0.1.152.0            NuGetGallery This package contains reusable powershell automation for ex…
CloudTek.Automation.K8S   0.1.68.0             NuGetGallery This package contains reusable kubernetes powershell automa…
CloudTek.Build            0.11.3.0             NuGetGallery Package Description

Installing explicitly from that repo fails as well

  Install-PSResource -RequiredResource @{
    "$($_.ModuleName)" = @{
      version = "$($_.ModuleVersion)"
      repository = "$($_.ModuleName)-local"
     }
  } -Verbose -Debug;

Are there any preconditions I should be aware of?

• Are there any limitations on the repository name?
• What is the expected value for the Uri in the repo registration? Examples on the web indicated that an absolute path to a folder should be acceptable.
• What is the expected folder & file structure of the target folder? (I think this is the most important question to ask)

# $path = "/home/runner/work/automation/automation/src/CloudTek.Automation.Shell"
function Register-LocalPSResourceRepository([string] $path, [string]$name) {
  [string]$p = Resolve-Path -Path $path;
  Write-Host "Registering PSRepository ($name : $p) ..." -ForegroundColor Gray;
  [hashtable[]]$repositories = @(
    @{ Name = $name; Uri = "$p"; Trusted = $true; Priority = 10 }
  );

  Register-PSResourceRepository -Repository $repositories -Verbose;
}

[anamnavi]

@mmisztal1980 thanks for raising these questions, I will work on making the documentation clear for such examples/limitations.

• When performing the Find operation against a local repository, the package name cannot contain a wildcard. This is a limitation with local repository search functionality. So you would run Find-PSResource -Name "CloudTek.Automation.Shell" -Repository <yourLocalRepo>
• For a local repository's registration the Uri can be an absolute or relative path.

The intended flow when finding a package from a local repository is as follows:

1. Register the local repository. This will be a folder where package .nupkg files are stored. (Register-PSResourceRepository -Name "MyLocalRepo" -Uri "C:\Users\johndoe\Documents\RepoFolder")
2. Ensure the .nupkg file for the package is present at this location. You could either manually move the .nupkg file to the repository path or you can use Save-PSResource which basically downloads the package from a repository to a location of your specified choice. (Save-PSResource -Name "CloudTek.Automation.Shell" -Repository NuGetGallery -Path "C:\Users\johndoe\Documents\RepoFolder" -AsNupkg). The -AsNupkg is important otherwise it'll download and extract the contents of the package and you'll have a folder of contents in the repository path, but you want the .nupkg file.
3. Now you can search for the package from your local repository. (Find-PSResource -Name "CloudTek.Automation.Shell" -Repository MyLocalRepo). Because a .nupkg file is present at the local repository path it will find that package in your local repository.

So to answer your 3rd question the expected structure of your local repository's path is that it should contain .nupkg file for the package you wish to find there. If your local repository should have 10 packages there would be 10 .nupkg files. Let me know if this helps