[Question]Getting error while running the extractor pipline

[abhijeetdhoke]

Release version

Not sure

Question Details

While run extractor task which in the extractor pipeline i am getting below error
I am using git repository but for pipeline we are using azure DevOps pipeline.

Below is script :
parameters:

• name: APIM_INSTANCE_NAME
  displayName: APIM instance name
  type: string
  default: apim-eus2-01
• name: RESOURCE_GROUP_NAME
  displayName: APIM instance resource group name
  type: string
  default: rg-eus2-01
• name: APIM_REPOSITORY_NAME
  type: string
  displayName: APIM repository for pull request
  default: POCAPIOps
• name: API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH
  type: string
  displayName: Folder where you want to extract the artifacts
  default: artifacts
• name: TARGET_BRANCH_NAME
  type: string
  displayName: Target branch for pull request
  default: main
• name: CONFIGURATION_YAML_PATH
  type: string
  displayName: Optional configuration file
  values:
  • Extract All
  • configuration.extractor.yaml
• name: API_SPECIFICATION_FORMAT
  type: string
  displayName: API Specification Format
  values:
  • OpenAPIV3Yaml
  • OpenAPIV3Json
  • OpenAPIV2Yaml
  • OpenAPIV2Json

trigger: none

variables:

• group: apim-automation
• name: System.Debug
  value: true

stages:

• stage: create_artifact_from_portal
  displayName: Create artifact from portal
  jobs:

  • job: create_artifact_from_portal
    displayName: Create artifact from portal
    pool:
    pool:
    name: CodeWay2
    steps:
    - powershell: |
    Write-Output "Checking Azure CLI installation..."
    $ErrorActionPreference = "SilentlyContinue"
    $CLIInstalled = az --version
    $ErrorActionPreference = "Stop"
    if (!$CLIInstalled) {
    Write-Output "Azure CLI not found. Starting installation..."
    Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile AzureCLI.msi
    Write-Output "Downloaded Azure CLI installer."
    Start-Process msiexec.exe -ArgumentList "/i AzureCLI.msi /quiet" -Wait
    # Update the PATH for the current session
    $AzureCLIPath = "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin" # Default installation path
    $env:Path += ";$AzureCLIPath"
    Write-Output "Azure CLI installation completed and path updated for current session."
    } else {
    Write-Output "Azure CLI is already installed."
    }
    Write-Output "Listing all environment variables..."
    Get-ChildItem env: | Format-Table Name, Value -AutoSize | Out-String -Width 2048
    displayName: 'Install Azure CLI and List Env Variables'
    - task: AzurePowerShell@5
    displayName: Set extraction variables
    inputs:
    azureSubscription: 'cdax-sa-azure-dev'
    ScriptType: 'InlineScript'
    Inline: |
    Set-StrictMode -Version Latest
    $ErrorActionPreference = "Stop"
    $VerbosePreference = "Continue"
    $InformationPreference = "Continue"
    Write-Host "Extracting Azure Bearer Token..."
    $accessToken = & "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az" account get-access-token --query "accessToken" --output tsv
    Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_BEARER_TOKEN]$accessToken"
    Write-Host "Setting Azure Client ID, Client Secret, and Tenant ID..."
    Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_ID]$(servicePrincipalId)"
    Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_CLIENT_SECRET]$(servicePrincipalKey)"
    Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_TENANT_ID]$(tenantId)"
    Write-Host "Checking Azure Subscription ID..."
    if (-not $env:AZURE_SUBSCRIPTION_ID) {
    $subscriptionCount = & "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az" account list --query "length([])" --output tsv
    if ($subscriptionCount -eq 1) {
    $subscriptionId = & "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az" account list --query "[0].id" --output tsv
    Write-Host "Setting AZURE_SUBSCRIPTION_ID environment variable to: $subscriptionId"
    Write-Host "##vso[task.setvariable issecret=true;variable=AZURE_SUBSCRIPTION_ID]$subscriptionId"
    } elseif ($subscriptionCount -gt 1) {
    Write-Host "Multiple subscriptions are accessible. Please set the AZURE_SUBSCRIPTION_ID environment variable manually."
    exit 1
    }
    } else {
    Write-Host "AZURE_SUBSCRIPTION_ID is already set to: $env:AZURE_SUBSCRIPTION_ID"
    }
    azurePowerShellVersion: 'LatestVersion'
    addSpnToEnvironment: true
    failOnStandardError: true
    - task: PowerShell@2
    displayName: Fetch extractor
    inputs:
    targetType: "inline"
    script: |
    Set-StrictMode -Version Latest
    $ErrorActionPreference = "Stop"
    $VerbosePreference = "Continue"
    $InformationPreference = "Continue"

          Write-Information "Downloading extractor..."
          $extractorFileName = "extractor.linux-x64"
          $extractorFinalFileName = "extractor"
        
          if ("$(Agent.OS)" -like "*win*") {
            $extractorFileName = "extractor.win-x64.exe"
            $extractorFinalFileName = "extractor.exe"
          }
          elseif ("$(Agent.OS)" -like "*mac*" -and "$(Agent.OSArchitecture)" -like "*arm*") {
            $extractorFileName = "extractor.osx-arm64"
          }
          
          $uri = "https://github.com/Azure/apiops/releases/download/$(apiops_release_version)/$extractorFileName"
          $destinationFilePath = Join-Path "$(Agent.TempDirectory)" $extractorFinalFileName
          Invoke-WebRequest -Uri "$uri" -OutFile "$destinationFilePath"
    
          if ("$(Agent.OS)" -like "*linux*")
          {
            Write-Information "Setting file permissions..."
            & chmod +x "$destinationFilePath"
            if ($LASTEXITCODE -ne 0) { throw "Setting file permissions failed."}
          }
    
          Write-Host "##vso[task.setvariable variable=EXTRACTOR_FILE_PATH]$destinationFilePath"
          Write-Information "Execution complete."
        failOnStderr: true
        pwsh: true
    - task: PowerShell@2
      displayName: Run extractor
      inputs:
        targetType: "inline"
        script: |
          Set-StrictMode -Version Latest
          $ErrorActionPreference = "Stop"
          $VerbosePreference = "Continue"
          $InformationPreference = "Continue"
    
          if (Test-Path -Path "$(EXTRACTOR_FILE_PATH)") {
            & "$(EXTRACTOR_FILE_PATH)"
            if ($LASTEXITCODE -ne 0) { throw "Running extractor failed with exit code: $LASTEXITCODE"}
          } else {
            throw "Extractor file not found at $(EXTRACTOR_FILE_PATH)"
          }
    
          Write-Information "Execution complete."
        failOnStderr: true
        pwsh: true
      env:
        AZURE_BEARER_TOKEN: $(AZURE_BEARER_TOKEN)
        AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
        AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)
        AZURE_TENANT_ID: $(AZURE_TENANT_ID)
        AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
        AZURE_RESOURCE_GROUP_NAME: ${{ parameters.RESOURCE_GROUP_NAME }}
        API_MANAGEMENT_SERVICE_NAME: ${{ parameters.APIM_INSTANCE_NAME }}
        API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: $(Build.ArtifactStagingDirectory)/${{ parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}
        API_SPECIFICATION_FORMAT: ${{ parameters.API_SPECIFICATION_FORMAT }}
        ${{ if ne( parameters['CONFIGURATION_YAML_PATH'], 'Extract All' ) }}:
          CONFIGURATION_YAML_PATH: ${{ parameters.CONFIGURATION_YAML_PATH }}
    #Running Super Linting Tool on the API(s) - START
    

---

##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '-NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent_work_temp\34a2c3fe-3445-47dd-93cd-4161c48a3988.ps1'"'
##[debug] FileName: 'C:\Program Files\PowerShell\7\pwsh.exe'
##[debug] WorkingDirectory: 'C:\agent_work\6\s'
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent_work_temp\34a2c3fe-3445-47dd-93cd-4161c48a3988.ps1'"
##[debug]STDERR: Unhandled exception. System.ArgumentNullException: Value cannot be null. (Parameter 'jwtEncodedString')
##[debug]STDERR: at Microsoft.IdentityModel.JsonWebTokens.JsonWebToken..ctor(String jwtEncodedString)
##[debug]STDERR: at extractor.Program.GetCredentialFromToken(String token)
##[debug]STDERR: at extractor.Program.GetTokenCredential(IConfiguration configuration)
##[debug]STDERR: at extractor.Program.GetAuthenticatedHttpPipeline(IServiceProvider provider)
##[debug]STDERR: at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
##[debug]STDERR: at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) ##[debug]STDERR: at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context) ##[debug]STDERR: at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
##[debug]STDERR: at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)

Expected behavior

Extractor pipeline should run

Actual behavior

getting error in extractor pipeline

Reproduction Steps

1. I have created repo in github
2. added all yml file
3. connected to azure devOps
4. try to run the extractor pipeline.
5. getting error

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[waelkdouh]

This error seems to be specific to your environment.

[abhijeetdhoke]

@waelkdouh Thank you for your response. While I appreciate the insight, I'm seeking more specific details to address the issues we're encountering. For example, could you provide further clarification on the exact issues with the self-hosted agent and where exactly ? This would greatly assist us in identifying and resolving the problem more effectively.

[guythetechie]

@abhijeetdhoke - the error suggests that it's not finding an authentication token in the environment variable AZURE_BEARER_TOKEN. It should have been set here. Make sure that step ran successfully and set the environment variable.

[abhijeetdhoke]

@guythetechie @waelkdouh "Thank you for your response. I've managed to figure it out, thanks. However, a question arises regarding the safety of using extractor.exe from a client perspective. Could you please suggest how I can effectively argue in favor of its safety?"

[waelkdouh]

@abhijeetdhoke always happy to help. To answer your question you can share with them that they are liberty to take the source code from our repo and compile it themselves. It's all open source. Just remember to have them update their pipelines to fetch the executables from their own feed (e.g azure devops feed).