Error when importing application using Jenkins

[emmanuel-galindo]

API-Manager CLI Version
1.3.11

API-Manager and Service-Pack Version
7.7 march

Question
Getting an error we can't reproduce. We don't know what triggers it but at some point (not very often but when it happens devs needs to get support from admins to resolve it, so it is inconvenient) we got an error when trying to import an update for an existing api and a linked app

0      INFO     APIManagerAdapter| Successfully connected to API-Manager (7.7.20210330) on: https://api-manager-dev.*.*.com:****
829    INFO  IImportConfigAdapter| Reading API-Definition (Swagger/WSDL) from file: '/tmp/workspace/cd/system-api-master/api/System-API.v1.json' (absolute path)
870    INFO      APISpecification| Used the backendBasepath: 'https://www.test.com/api' to adjust the API-Specification.
879    INFO  IImportConfigAdapter| Loading image from: '/tmp/workspace/cd/system-api-master/api/config/image.png'
1014   INFO  IImportConfigAdapter| Handling configured client-applications.
1829   INFO  IImportConfigAdapter| Found existing application: 'app' (null) based on given name 'app'
7606   INFO  APIManagerAPIAdapter| Found existing API on path: '/api' (unpublished) (ID: '7464c602-20c3-40a5-9d40-c753b26d7854'
7608   INFO      APISpecification| Detected API-Definition-Filesizes: API-Manager: 51786 vs. Import: 24844
7780   INFO      APIImportManager| Recognized the following changes. Potentially Breaking: [apiDefinition, securityProfiles] plus Non-Breaking: [version]
7780   INFO      APIImportManager| Update API Strategy: Re-Create API as changes can't be applied to existing API.
7780   INFO   RecreateToUpdateAPI| Create new API to update existing: 'System API' (ID: 7464c602-20c3-40a5-9d40-c753b26d7854)
8065   INFO          CreateNewAPI| Create unpublished API: 'System API' 1.6.2 based on Swagger 2.0 specification.
9921   INFO      ManageClientApps| Creating API-Access for the following apps: '[[app]]'
9921   INFO      ManageClientApps| Creating API-Access for application 'app'
10177  WARN  agerAPIAccessAdapter| Got unexpected error: 'Unknown API' while creating API-Access ... Try again in 1 second.
11300  ERROR agerAPIAccessAdapter| Error creating/updating API Access: APIAccess [apiName=null, apiVersion=null, id=null, apiId=13c35b7c-1b37-4234-920c-ae9baad2e19e]. Response-Code: 404. Got response: '{"errors":[{"code":404,"message":"Unknown API"}]}'
11303  INFO        RollbackAction| Rollback FE-API: 'System API' (ID: '13c35b7c-1b37-4234-920c-ae9baad2e19e' / State: 'unpublished')
11559  INFO  APIManagerAPIAdapter| API: System API 1.6.2 (13c35b7c-1b37-4234-920c-ae9baad2e19e) successfully deleted
14258  INFO       RollbackHandler| Rolled back: '[Frontend-API: false, Backend-API: true]'
14259  ERROR         APIImportApp| Can't create API access requests.
                                 | Failure creating API-Access for application: 'app'. Error creating/updating API Access.
                                 | Error creating/updating API Access.
                                 | Error creating/updating API Access. Response-Code: 404

The error appear only when executed by jenkins in a pipeline. In there there's a gradle task emulating the mvn calls documented. That works most of the time.
But any subsequent build will keep failing.

Until the same gradle is executed in a workstation and that fixes the issue. Further CD builds complete again.

Today, after fixing the version in the config.json for an API, it triggered the error again. Below is the complete example.

What I've tried so far
Any idea, how can enable debug?

Additional information

Starting process 'command '/usr/lib/jvm/java-11-openjdk-11.0.6.10-3.el7.x86_64/bin/java''. Working directory: /tmp/workspace/cd/system-api-master Command: /usr/lib/jvm/java-11-openjdk-11.0.6.10-3.el7.x86_64/bin/java -Dfile.encoding=US-ASCII -Duser.country=US -Duser.language=en -Duser.variant -cp /home/****/.gradle/caches/modules-2/files-2.1/de.undercouch/gradle-download-task/4.0.4/1cf47e5d4497e61634226d049c0ef971426fa9b6/gradle-download-task-4.0.4.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.github.axway-api-management-plus.apim-cli/apimcli-apis/1.3.11/dbb0d7ad96973486b4df6691d8c5de3164434204/apimcli-apis-1.3.11.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.github.axway-api-management-plus.apim-cli/apimcli-tool/1.3.11/821af88cb79c1f3ad59d2e719a6a03e7730f8349/apimcli-tool-1.3.11.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.github.axway-api-management-plus.apim-cli/apimcli-apim-adapter/1.3.11/52b331319887aefef36bd266dae0a222774b31fc/apimcli-apim-adapter-1.3.11.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpmime/4.5.13/efc110bad4a0d45cda7858e6beee1d8a8313da5a/httpmime-4.5.13.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpclient/4.5.13/e5f6cae5ca7ecaac1ec2827a9e2d65ae2869cada/httpclient-4.5.13.jar:/home/****/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/home/****/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.8.0/92999e26e6534606b5678014e66948286298a35c/commons-io-2.8.0.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-text/1.8/879a6bde4c0537a25504c72ec7a94ba4099f469c/commons-text-1.8.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/2.11.3/4e3bc00e0175c9be7bbe41c9c6b000591d0a2567/jackson-dataformat-yaml-2.11.3.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.11.3/4f7b27416934dc929bb6c2d2c5fe521829e6a4ec/jackson-databind-2.11.3.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.11.3/c2351800432bdbdd8284c3f5a7f0782a352aa84a/jackson-core-2.11.3.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.30/c21f55139d8141d2231214fb1feaf50a1edca95e/slf4j-log4j12-1.7.30.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.github.freva/ascii-table/1.1.0/febb5cb3665e979d84e2fa5c97c176a2d1e8376b/ascii-table-1.1.0.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-csv/1.8/37ca9a9aa2d4be2599e55506a6d3170dd7a3df4/commons-csv-1.8.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-lang3/3.11/68e9a6adf7cf8eb7e9d31bbc554c7c75eeaac568/commons-lang3-3.11.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.ehcache/ehcache/3.8.1/ae19c7f2f63f5e88f00ccd189b22c535ab75b01a/ehcache-3.8.1.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.apache.httpcomponents/httpcore/4.4.13/853b96d3afbb7bf8cc303fe27ee96836a10c1834/httpcore-4.4.13.jar:/home/****/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.2/4bfc12adfe4842bf07b657f0369c4cb522955686/commons-logging-1.2.jar:/home/****/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.11/3acb4705652e16236558f0f4f2192cc33c3bd189/commons-codec-1.11.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.11.3/25d4e9c777e7a8805c4a000a8629d3009c779c9b/jackson-annotations-2.11.3.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.30/b5a4b6d16ab13e34a88fae84c35cd5d68cac922c/slf4j-api-1.7.30.jar:/home/****/.gradle/caches/modules-2/files-2.1/log4j/log4j/1.2.17/5af35056b4d257e4b64b9e8069c0746e8b08629f/log4j-1.2.17.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.26/a78a8747147d2c5807683e76ec2b633e95c14fe9/snakeyaml-1.26.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.glassfish.jaxb/jaxb-runtime/2.3.1/dd6dda9da676a54c5b36ca2806ff95ee017d8738/jaxb-runtime-2.3.1.jar:/home/****/.gradle/caches/modules-2/files-2.1/javax.xml.bind/jaxb-api/2.3.1/8531ad5ac454cc2deb9d4d32c40c4d7451939b5d/jaxb-api-2.3.1.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.glassfish.jaxb/txw2/2.3.1/a09d2c48d3285f206fafbffe0e50619284e92126/txw2-2.3.1.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.sun.istack/istack-commons-runtime/3.0.7/c197c86ceec7318b1284bffb49b54226ca774003/istack-commons-runtime-3.0.7.jar:/home/****/.gradle/caches/modules-2/files-2.1/org.jvnet.staxex/stax-ex/1.8/8cc35f73da321c29973191f2cf143d29d26a1df7/stax-ex-1.8.jar:/home/****/.gradle/caches/modules-2/files-2.1/com.sun.xml.fastinfoset/FastInfoset/1.2.15/bb7b7ec0379982b97c62cd17465cb6d9155f68e8/FastInfoset-1.2.15.jar:/home/****/.gradle/caches/modules-2/files-2.1/javax.activation/javax.activation-api/1.2.0/85262acf3ca9816f9537ca47d5adeabaead7cb16/javax.activation-api-1.2.0.jar com.axway.apim.APIImportApp -c ./api/System-API.v1.config/config.json -h api-manager-dev.*.*.com -u **** -p **** -f true
0      INFO     APIManagerAdapter| Successfully connected to API-Manager (7.7.20210330) on: https://api-manager-dev.*.*.com:****
671    INFO  IImportConfigAdapter| Reading API-Definition (Swagger/WSDL) from file: '/tmp/workspace/cd/system-api-master/api/System-API.v1.json' (absolute path)
698    INFO      APISpecification| Used the backendBasepath: 'https://\*.\*.com/api' to adjust the API-Specification.
769    INFO  IImportConfigAdapter| Loading image from: '/tmp/workspace/cd/system-api-master/api/System-API.v1.config/image.png'
905    INFO  IImportConfigAdapter| Handling configured client-applications.
1783   INFO  IImportConfigAdapter| Found existing application: 'app' (null) based on given name 'app'
8461   INFO  APIManagerAPIAdapter| Found existing API on path: '/api' (unpublished) (ID: 'c5bbf2ef-2c61-458b-8964-6be4ec29c915'
8463   INFO      APISpecification| Detected API-Definition-Filesizes: API-Manager: 24846 vs. Import: 24846
8787   INFO      APIImportManager| Recognized the following changes. Potentially Breaking: [apiDefinition] plus Non-Breaking: [version]
8787   INFO      APIImportManager| Update API Strategy: Re-Create API as changes can't be applied to existing API.
8788   INFO   RecreateToUpdateAPI| Create new API to update existing: 'System API' (ID: c5bbf2ef-2c61-458b-8964-6be4ec29c915)
9124   INFO          CreateNewAPI| Create unpublished API: 'System API' 1.6.3 based on Swagger 2.0 specification.
10862  INFO      ManageClientApps| Creating API-Access for the following apps: '[[app]]'
10863  INFO      ManageClientApps| Creating API-Access for application 'app'
11130  WARN  agerAPIAccessAdapter| Got unexpected error: 'Unknown API' while creating API-Access ... Try again in 1 second.
12258  ERROR agerAPIAccessAdapter| Error creating/updating API Access: APIAccess [apiName=null, apiVersion=null, id=null, apiId=aa1df0b0-56e6-4c8a-96f2-a9f4440fd0a6]. Response-Code: 404. Got response: '{"errors":[{"code":404,"message":"Unknown API"}]}'
12264  INFO        RollbackAction| Rollback FE-API: 'System API' (ID: 'aa1df0b0-56e6-4c8a-96f2-a9f4440fd0a6' / State: 'unpublished')
12515  INFO  APIManagerAPIAdapter| API: System API 1.6.3 (aa1df0b0-56e6-4c8a-96f2-a9f4440fd0a6) successfully deleted
16291  INFO       RollbackHandler| Rolled back: '[Frontend-API: false, Backend-API: true]'
16293  ERROR         APIImportApp| Can't create API access requests.
                                 | Failure creating API-Access for application: 'app'. Error creating/updating API Access.
                                 | Error creating/updating API Access.
                                 | Error creating/updating API Access. Response-Code: 404

[cwiechmann]

Hi @emmanuel-galindo, As you may already guess by the log output, the behavior is not new to me, as the CLI is trying it again in 1 second. This tries to mitigate the issue.
The issue also appears sometimes, when the CLI is running in the Integration-Tests. My assumption is, that the API-Manager (maybe Cassandra) is taking a moment to replicate the information.

To be clear what should happen:
The new API is created with a certain Frontend-API-ID. This ID is used to created an API-Access object. Like the following:

{"apiId":"a3287b08-8fed-4101-9ff8-710f0372d622","enabled":true}

And this is send with a POST-Request. e.g.

POST /api/portal/v1.4/applications/d33bc178-d686-41c4-a9b0-8b5a0eabbc5d/apis

to the API-Manager. From this point on, the CLI has no control anymore. The API-Manager internally now comes to the conclusion, the API with the given ID is unknown. But of course, it is not.

Three questions:

• Is the APIM-CLI talking to a Load-Balanced API-Manager?
• How many APIs do you have ap. in the catalog? (maybe that replication takes longer, the more APIs you have)
• Do you have enabled the data-caching feature for the API-Manager?
  <VMArg name="-Dcom.axway.apimanager.api.data.cache=true"/>

Until the same gradle is executed in a workstation and that fixes the issue. Further CD builds complete again.

It's not clear to my, how this solves the issue?

[cwiechmann]

Maybe we have to wait longer than only 1 second.

[emmanuel-galindo]

@cwiechmann wow.. thank you for the quick response. It is reassuring that we picked the right tool for our cd/ci needs.

About how running it from the workstation fixes it.. I don't know, this is empirical.

For the questions

• Is the APIM-CLI talking to a Load-Balanced API-Manager?
  No, is a single instance.

• How many APIs do you have ap. in the catalog? (maybe that replication takes longer, the more APIs you have)
  Around 230 feAPIs

• Do you have enabled the data-caching feature for the API-Manager?

I've opened # 01291144 -- you might get more details about the implementation there.

[emmanuel-galindo]

For the remaining question around data caching, that property is not set.

[cwiechmann]

I would recommend for now, to make this Retry-Timeout configurable instead of the hardcoded 1 second, as I believe the API-Manager internally has not yet synchronized/replicated the just registered API.

What do you think @emmanuel-galindo?

[emmanuel-galindo]

@cwiechmann
that sounds great,

maybe another option for the future, also a way to to configure the qty of retries? because maybe 1 is good.. but then to have the option to try 50 times until it sticks

[cwiechmann]

Hi @emmanuel-galindo, I have created the release 1.3.12 which supports a new parameter: -retryDelay:

# You may set the retryDelay parameter if you experience quite often unexpected responses from the 
# API-Manager REST-API. For instance, reporting an Unknown API for an API just created, while trying 
# to create access for it. 
# This may occur more often, when you have more APIs, Applications, etc. in your API-Manager registry.
# The APIM-CLI will try again such requests if the response is unexpected. As all other parameters you 
# can also set it as a command line parameter: -retryDelay 3000. See issue: #213 for more details.
# Defaults to 1000 (1 second)
# Example delay of 2 seconds: 
# retryDelay=2000

Can you check, if this helps to solve the problem?

[emmanuel-galindo]

@cwiechmann great! thanks for the quick turnaround

just added it to the pipeline with 10secs. and we can't reproduce it so I am closing this ticket!

thanks!