[BUG] Extractor pipeline not honouring configuration.extractor.yaml

[vinp007]

Release version

4.1.0

Describe the bug

Extractor pipeline not honouring the API names mentioned in the configuration.extractor.yaml

Expected behavior

Extractor pipeline only extract APIs matching with the API names mentioned in the configuration.extractor.yaml

Actual behavior

Extractor pipeline is extracting all the APIs and ignoring the contents of the configuration.extractor.yaml.

When configuration.extractor.yaml option is selected while running the extractor pipeline, there are 2 scenarios -

1. When configuration.extractor.yaml is absent - the pipeline execution fails and the pipeline logs clearly states that the pipleine has failed as it can't find the configuration.extractor.yaml.

2. When configuration.extractor.yaml is present - The pipeline execution seems to be ignoring it altogether and instead extracting all the APIs. Also there is no mention of anything related to configuration.extractor.yaml in the pipeline logs whatsoever even when using trace logging level.

Reproduction Steps

Specify couple of API names in the configuration.extractor.yaml and run the extractor pipeline

[waelkdouh]

Couldn't reproduce on our side. Can you share some logs? Make sure you scrub the logs.

[vinp007]

Thanks for the reply. Sorry it's very difficult to remove all the IP info from the logs to sanitise it without losing the useful content. Also, there isn't anything captured in the logs about the configuration.extractor.yaml so I doubt that would help.

Would someone be able to check the sample configuration.extractor.yaml file which is part of the code on Github and confirm if there isn't any typo anywhere in the tag name i.e. apiNames: and there isn't any issue with indentation. Also the api1 and api2 mentioned, are they the api names or API display names?

Have also looked at the extractor pipeline code and couldn't find any reference of how the configuration.extractor.yaml is being used in the pipeline. Would be great to know how it has been linked to the pipeline.

Thanks again

[waelkdouh]

Here is a successful run

and here is the configuration file

[shawnmurtagh]

It might help to share/inspect your run-extractor.yaml file in case the variable references are causing the extractor to find no configuration file in this step:

[Kelele-m3]

Hi @waelkdouh, I was extracting the apim resources from the portal, & the only extracted contents were the "APIs" & "products" but I was only expecting the other contents to be extracted like the contents here: #297 (comment)

any idea what I a missing?

[waelkdouh]

Can you share the logs here? Make sure to scrub them first.

[Kelele-m3]

Can you share the logs here? Make sure to scrub them first.

how can in include the other folders other than apis & products in the extraction?

[Kelele-m3]

Only getting these files in the extraction output:

[waelkdouh]

It should happen automatically. This is why I'm asking you for the extractor Ron logs to see why this did not happen automatically.

[waelkdouh]

Remember that the extractor is using git diffing so basically every time you run the extractor, it runs against the API instance, and brings down everything that is different than what you already have in your git repository.

[Kelele-m3]

well that was my expectation but i ain't getting them all

[Kelele-m3]

It should happen automatically. This is why I'm asking you for the extractor Ron logs to see why this did not happen automatically.

2023-06-21T13:05:13.9370590Z ##[debug]Evaluating condition for step: 'Download pipeline artifact'
2023-06-21T13:05:13.9372038Z ##[debug]Evaluating: SucceededNode()
2023-06-21T13:05:13.9372232Z ##[debug]Evaluating SucceededNode:
2023-06-21T13:05:13.9372712Z ##[debug]=> True
2023-06-21T13:05:13.9372965Z ##[debug]Result: True
2023-06-21T13:05:13.9373185Z ##[section]Starting: Download pipeline artifact
2023-06-21T13:05:13.9476285Z ==============================================================================
2023-06-21T13:05:13.9476405Z Task : Download Pipeline Artifacts
2023-06-21T13:05:13.9476469Z Description : Download build and pipeline artifacts
2023-06-21T13:05:13.9476555Z Version : 2.198.0
2023-06-21T13:05:13.9476603Z Author : Microsoft Corporation
2023-06-21T13:05:13.9476661Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/download-pipeline-artifact
2023-06-21T13:05:13.9476762Z ==============================================================================
2023-06-21T13:05:14.2800146Z ##[debug]TargetPath: D:\a\1/artifacts-from-portal
2023-06-21T13:05:14.2827813Z ##[debug]ArtifactName: artifacts-from-portal
2023-06-21T13:05:14.2847813Z ##[debug]BuildResult: Succeeded
2023-06-21T13:05:14.2848270Z ##[debug]Run: CurrentRun
2023-06-21T13:05:14.2848643Z ##[debug]ProjectId: 772b1596-c97f-40e7-bd1b-72c1ae8dcff2
2023-06-21T13:05:14.2864313Z Download from the specified build: #119
2023-06-21T13:05:14.2877174Z ##[debug]Processed: ##vso[task.setvariable variable=BuildNumber;issecret=False;]119
2023-06-21T13:05:14.2877770Z Download artifact to: D:\a\1/artifacts-from-portal
2023-06-21T13:05:14.6395366Z Using default max parallelism.
2023-06-21T13:05:14.6412690Z Max dedup parallelism: 192
2023-06-21T13:05:14.9021256Z Verbose, ArtifactHttpClientFactory.CreateVssHttpClient: DedupStoreHttpClient with BaseUri: https://vsblobprodweu2.vsblob.visualstudio.com/A8a9ffe2e-8531-416e-b562-59f9c3f3576e/, MaxRetries:5, SendTimeout:00:05:00
2023-06-21T13:05:15.0156751Z Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 5097b5a9-f8a6-463c-8823-abf3450986b5
2023-06-21T13:05:15.0157350Z Verbose, Started ITelemetrySender BlobStoreApplicationInsightsTelemetrySender.
2023-06-21T13:05:15.0256589Z Verbose, PipelineArtifact.DownloadMultipleArtifactsAsync starting
2023-06-21T13:05:15.0347004Z Information, DedupManifestArtifactClient will correlate http requests with X-TFS-Session 5097b5a9-f8a6-463c-8823-abf3450986b5
2023-06-21T13:05:15.0364714Z Information, Minimatch patterns: [**]
2023-06-21T13:05:15.3887106Z Information, Filtered 16 files from the Minimatch filters supplied.
2023-06-21T13:05:15.4198442Z Verbose, Could not initialize dataport: Could not initialize dataport for 'D:' as 'Class not registered: E_CLASSNOTREG 80040154'
2023-06-21T13:05:15.4230410Z Information, Downloaded 0.0 MB out of 1.0 MB (0%).
2023-06-21T13:05:15.5774979Z Verbose, Downloaded /apim/products/starter/groups.json
2023-06-21T13:05:15.5775450Z Verbose, Downloaded /apim/products/starter/productInformation.json
2023-06-21T13:05:15.5808757Z Verbose, Downloaded /apim/products/testing-public-api/groups.json
2023-06-21T13:05:15.5808997Z Verbose, Downloaded /apim/products/starter/policy.xml
2023-06-21T13:05:15.5809193Z Verbose, Downloaded /apim/products/unlimited/productInformation.json
2023-06-21T13:05:15.5816425Z Verbose, Downloaded /apim/products/unlimited/groups.json
2023-06-21T13:05:15.5844874Z Verbose, Downloaded /apim/products/starter/apis.json
2023-06-21T13:05:15.5892147Z Verbose, Downloaded /apim/products/unlimited/apis.json
2023-06-21T13:05:15.5897379Z Verbose, Downloaded /apim/tags/University/tagInformation.json
2023-06-21T13:05:15.5916313Z Verbose, Downloaded /apim/apis/usa-universities/apiInformation.json
2023-06-21T13:05:15.5920837Z Verbose, Downloaded /apim/products/testing-public-api/productInformation.json
2023-06-21T13:05:15.5937981Z Verbose, Downloaded /apim/policy.xml
2023-06-21T13:05:15.5993129Z Verbose, Downloaded /apim/tags/apiEntries/tagInformation.json
2023-06-21T13:05:15.6044656Z Verbose, Downloaded /spectral-result.xml
2023-06-21T13:05:15.6063881Z Verbose, Downloaded /apim/apis/usa-universities/operations/getAllUniversities/policy.xml
2023-06-21T13:05:15.8437671Z Verbose, Downloaded /apim/apis/usa-universities/specification.json
2023-06-21T13:05:15.8461848Z Information, Downloaded 1.0 MB out of 1.0 MB (100%).
2023-06-21T13:05:15.8470045Z Information,
2023-06-21T13:05:15.8470242Z Download statistics:
2023-06-21T13:05:15.8470507Z Total Content: 1.0 MB
2023-06-21T13:05:15.8470773Z Physical Content Downloaded: 0.2 MB
2023-06-21T13:05:15.8470993Z Compression Saved: 0.8 MB
2023-06-21T13:05:15.8471218Z Local Caching Saved: 0.0 MB
2023-06-21T13:05:15.8471429Z Chunks Downloaded: 27
2023-06-21T13:05:15.8471632Z Nodes Downloaded: 0
2023-06-21T13:05:15.8471794Z
2023-06-21T13:05:15.8585567Z Information, Download completed.
2023-06-21T13:05:15.8587399Z Verbose, PipelineArtifact.DownloadMultipleArtifactsAsync completed in 0.8269002 seconds
2023-06-21T13:05:15.8772363Z ##[debug]Processed: ##vso[telemetry.publish area=AzurePipelinesAgent;feature=PipelineArtifact]{"FileCount":"0","PlanId":"5c5f53a3-5267-42db-b81e-8e98e941a399","JobId":"cb2c978d-9a82-5791-38d8-088210f09300","TaskInstanceId":"175864ef-cfd2-5523-fe92-2d9eada74d7a","ActionDurationMs":"826","ActionName":"PipelineArtifact.DownloadMultipleArtifactsAsync","ActionResult":"Success","AttemptNumber":"1","ItemCount":"0","Level":"ThirdParty","CreatedUtcNow":"2023-06-21T13:05:15.0223545Z","SentUtcNow":"2023-06-21T13:05:15.8533948Z","BaseAddress":"https://vsblobprodweu2.vsblob.visualstudio.com/A8a9ffe2e-8531-416e-b562-59f9c3f3576e/","X_TFS_Session":"5097b5a9-f8a6-463c-8823-abf3450986b5","DeploymentEnvironment":"PRODUCTION","DeploymentEnvironmentIsProduction":"True","VSOAccount":"vsblobprodweu2","OSName":"Microsoft Windows","OSVersion":"10.0.20348","FrameworkDescription":".NET ","ProcessName":"Agent.PluginHost","DotNetReleaseDword":"-1","Version":"19.212.33005.0 built by: master (16b17fe44d)","ExceptionCount":"0"}
2023-06-21T13:05:15.8775136Z Verbose, ApplicationInsightsTelemetrySender waiting for 1 of 2 TrackEvent operations to complete...
2023-06-21T13:05:15.9361430Z Verbose, ApplicationInsightsTelemetrySender sent DedupDownload.DownloadAsync telemetry
2023-06-21T13:05:15.9383007Z Verbose, ApplicationInsightsTelemetrySender sent PipelineArtifact.DownloadMultipleArtifactsAsync telemetry
2023-06-21T13:05:15.9383673Z Verbose, ApplicationInsightsTelemetrySender operations completed in 79 ms
2023-06-21T13:05:15.9384554Z Information, ApplicationInsightsTelemetrySender correlated 2 events with X-TFS-Session 5097b5a9-f8a6-463c-8823-abf3450986b5
2023-06-21T13:05:15.9385168Z Verbose, Stopped ITelemetrySender BlobStoreApplicationInsightsTelemetrySender.
2023-06-21T13:05:15.9395190Z Downloading artifact finished.
2023-06-21T13:05:15.9515052Z ##[section]Finishing: Download pipeline artifact

[waelkdouh]

Do me a favor. Can you start with an empty repository and then run the extractor to extract everything?

[Kelele-m3]

Do me a favor. Can you start with an empty repository and then run the extractor to extract everything?

it is empty except for the extractor tools, & the newly extracted resources

[waelkdouh]

That is odd. It shouldn’t behave this way. Never had this come up before actually. Aside from double checking that you indeed have these resources that you are expecting as part of the apim instance itself not sure what else to check here. @guythetechie any thoughts?

[Kelele-m3]

@guythetechie any idea what is happening with my extractor missing some apim resources?

[waelkdouh]

@Kelele-m3 do you have the resources you are claiming are missing? for exampled namedvalues? Share some pictures from your azure portal if possible.

[guythetechie]

@Kelele-m3 - couple of thoughts:

1. Your configuration.extractor.yaml might be misconfigured. Can you share it?
2. Can you enable debug or trace logging? Your logs now will show the HTTP calls made to APIM during the extractor run. Make sure that HTTP calls are targeting the correct APIM instance.