Skip to content

APIGW Porting Notes

Jump to bottom
mmatsa edited this page Oct 25, 2019 · 8 revisions

APIGW Porting Notes

APIC - Two Gateway Options

Starting with version API Connect 2018, there are two options for Gateways.

(1) DataPower Gateway - v5 compatible

This Gateway is compatible with APIC v5, and therefore after the migration from v5 to v2018 there is no further need to modify the API.

(2) DataPower API Gateway

This is a new Gateway available in APIC 2018. It provides many improvements in interface, performance, debugability, etc. It is not fully backwards compatible. After migrating from APIC v5 to APIC v2018, you have the option of staying with the v5c Gateway or of porting your API to the apigw. If you choose to port your API to take advantage of the API Gateway features, this document has some helpful notes for things to think about during your porting process.

Path - v5 to APIGW

APIGW - High-Level Design

APIGW was redesigned and reimplemented "from scratch" to have improved interfaces for easier API Development, debug, performance, etc. In most ways it is compatible with v5c, but these improvements necessitate porting.

Here are some high-level examples (see below for lower-level details):

  • Programming model improved in non-compatible ways
    • v5c is request/response (old SOAP model), APIGW is "horseshoe" flow so natively supports 0/1/many backends
    • Explicit configuration - WYSIWYG - v5c had implicit behaviors which were hard to understand/configure/measure-performance - APIGW makes them explicit so that the user has a much easier time of understanding what's going on --- but that means they need to port
    • OAuth is usable in the assembly which gives much more programming flexibility but thus changes the way OAuth data is available in the API Context variables.
  • Performance improved in many non-compatible ways
    • almost every policy reimplemented "from scratch", which means that unspecified behavior is different. Users making use of APIC-defined behavior would be ok, but users making use of properties that depend on the underlying implementation would find them no longer available. In general, the documented data in the API Context is much superior to what had been available in v5c, however the undocumented accidental data is missing.
    • Streaming by default - but that means you don't have a buffered copy of data to work with unless you disable the streaming or explicitly save a copy in your assembly.
  • Many bugs fixed in non-compatible ways:
    • APIGW is much more spec-compliant, for example with the CORS and Badgerfish specifications, which means it's less conformant with v5c.
    • Lots of inconsistencies and bugs resolved. For example, some places where v5c had two ways of doing things with different behaviors in corner cases, APIGW has only one way which is explicitly configurable for the corner cases.
  • Security improved in many non-compatible ways
    • Code injection - v5c had multiple ways to do things, some of which allowed dynamic code from clients to be run, if the assembly author allowed it. APIGW leaves ways for all of that flexibility, but in ways that doesn't allow code injection even when the assembly author is using the dynamism.
    • Token management across the cluster which requires configuring a token-mgmt cluster separate from the GW-cluster. Native OAuth providers now contain their own personal token secret. Etc.

Porting to APIGW - Details

  • Gateway Type

    • To target the new APIGW, you have to set the gateway to datapower-api-gateway. Doing only that change would break because the policies used in the assembly need to all change, see below for more information about the other changes. However, this is a required step so that you are targeting the new Gateway.

  • Custom Policies

    • no support yet in APIGW

  • Gateway Extensions

    • support for APIGW for putting DP Config in Gateway Extensions, but none of the "plug points" to run it (pre-request, post-request, post-response and post-error).  Instead, Global Policies provides "plug points".  This does not replace full possibilities of Gateway Extensions in v5c, partially because some of those things are not applicable - for example v5c has "beginning of response rule" but APIGW has a "horseshoe" design with multiple first-class backend invokes supported, so there is no such thing as a "response rule".

  • Assembly Policies Support ("Out of the box" policies)

    • No policies of the same version are supported in both v5c and APIGW.
    • API Assemblies must be ported (not migrated) from v5c to APIGW.
    • Policy implementations do not necessarily yield identical results (for example invoke for v5c is not identical to invoke in APIGW).
    • For most v5c policies (e.g. Invoke 1.0.0) similar functionality is provided in APIGW (e.g. Invoke 2.0.0) but the major version is increased because the parameters can be different, the performance can be different, and the implementation can be completely different.  See other notes here for specifics on various policies, but this note is that no policy will have the same version number in v5c and APIGW, so as you update your assembly to configure each policy for the new usage you must increment the version number when it is ready for the new implementation.

  • Code Injection Protection

    • In v5c inline parameters were allowed on code, such as GWS, XSLT, and Switch conditions.  Since all of these have the ability to read from the API Context variables themselves, they do not require inline parameters for any of their functionality.  In APIGW we remove the ability to use inline parameters for them, which prevents code injection attacks while preserving all functional use cases.

  • Request Message

    • In APIGW, the request message is preserved to be exactly what came in, and instead modification such as injecting headers are only done on the mutable message message.  This also means that developers can not modify the request message.
    • In v5c, injected headers were also added to request.headers.

  • Parse Policy

    • The v5c implementation implicitly parses messages, but for the APIGW, every Assembly must explicitly parse wherever parsing is required.
    • Parse Policy is added in APIGW for explicit control of when parsing happens, which threat protection settings to apply to a parse, which kinds of input to allow, and how to gather metadata from the parse
    • APIGW has no implicit parsing - Assembly must explicitly parse when it desires parsing, which ensures that the correct limits and input checks are done.

  • Streaming

    • APIGW by default streams your data through the assembly flow, although many things you might need to do would force you to parse the input and thus break the streaming.
    • By contrast, v5c never streams the data and always parses and builds an in-memory representation of the input before running the assembly.
    • There is a toggle on the API to control this decision for the APIGW.

  • Activity Log - Analytics Integration

    • v5c provides an Activity-Log Policy that may be placed anywhere in the API Assembly. The actual Activity-Log policy takes no action other than setting the prescribed behavior that the Activity-Log implementation observes when it runs at the end of the assembly when it sends data to the gateway Analytics agent.
    • APIGW has moved these properties directly onto the API definition, so they appear in the Design Page of the API.  The same settings are allowed, but there is no Activity-Log Policy for the Assembly flow.
    • Because APIGW is streaming by default, you can not, by default, have error-content be payload, since that would require making a copy of the entire input payload in memory just in case there was a later error in the transaction.  If you want to log the entire payload in the event of error, you still can, but you would have to explicitly disable streaming.  In the UI, if you simply increase the log level, the UI will inform you that it needs to automatically disable streaming and allow you to confirm this decision without needing to visit the toggle separately.
    • In v5c, you could get the temporary activity-log settings in gws/xslt by using var://context/_apimgmt/analytics-type.  In APIGW, you must instead inspect the swagger/yaml, but they will be the actual activity-log settings that will end up being used and not only the ones set at that snapshot in time like in v5c.

  • Error responses

    • To determine whether to send back to the client an XML or JSON error, in APIGW the order is 1) the Accept header, 2) the Produce settings, and then 3) the request Content-Type header.   So produce overrides content-type, and the highest priority is Accept which overrides the others.
    • When the Accept HTTP header does not give a definitive answer as to the content type and schema of the error message to be returned (json, xml, etc), v5c will default to the last known setting of the content-type HTTP header.  A json request could be responded to with a xml error response in this case.  v6 will honor the original request content-type header in these cases.

  • Switch Policy

    • A single switch policy will replace three logic policies in v5c: 'switch', 'operation-switch' & 'if'
    • The way to specify each condition has changed.  In the v5c policies, one allowed a narrow DSL (domain specific language, specific to the operation-switch policy), one allowed JS, one allowed GWS.  In APIGW, all conditions are a version of JSONata with APIC extensions to allow for simple and easy conditions of the forms generally needed by APIC API developers. If you need a full GWS program to run, insert it before the switch.

  • Invoke Policy

    • No proxy policy exists in the APIGW (use the Invoke policy instead)
    • In v5c, proxy copies over URL parameters by default and Invoke does not.  These choices are all configurable in APIGW on Invoke.
    • In v5c, Proxy injected X-forwarded-* {protocol, hostname, port, etc.} headers, but Invoke did not.  In APIGW, this is configurable on the Invoke.

  • OAuth Policy and User Security Policy

    • OAuth policy and User Security policy are added
    • v5c OAuth 2 type API is removed.
    • A security object "OAuth provider settings" is created. This object gets referenced in the assembly policy "OAuth Token Generate" and also in the OAuth API Security.
    • User security policy can be used for user identity extraction, authentication and authorization. It is used by the OAuth provider object for enforcing resource owner security. In addition, user security policy can be used by any API in the assembly.
    • Crypto material on v5c is on the gateway service, whereas on APIGW it is on the oauth provider (on APIC, it can still be on gateway which gets copied over to all oauth providers)
    • For more details, see Security section below.

  • GatewayScript Policy (and XSLT Policy)

    • GatewayScript Policy provides a new improved interface for APIC integration, improved for ease of programming, consistency, more features, and underlying performance.
    • The new interface targets the new APIGW model, and therefore does not provide things like the session object which applies in the MPGW used by v5c, but not in APIGW.  (The APIGW only supports session.output.write for backward compatibility)
    • context module: https://www.ibm.com/support/knowledgecenter/en/SS9H2Y_7.7.0/com.ibm.dp.doc/context_apigw_js.html
    • multistep module: the multistep module works in both of APIGW and MPGW. It is used to execute a multistep rule. There is a table in this page to describe how the multistep module works differently in APIGW vs. MPGW : https://www.ibm.com/support/knowledgecenter/en/SS9H2Y_7.7.0/com.ibm.dp.doc/multistep_js.html
    • APIGW disables header-metadata module
    • If APIGW customers wish to use their GWS code from v5c without the benefits of the new interface, then there is a Compatibility Layer provided.  For use of this layer, the GWS must explicitly use var apim = require('apim'); to use the apim backward compatible APIs. Not all APIs are available, some porting might still be required.

  • XSLT Policy

  • Validate Policy

    • APIGW adds support for validating against XML Schema directly as well as the XML Schema in a WSDL.
    • Validate in v5c has a single "definition" property. Valid values for "definition" are request, response, the relative definition URL (#/definitions/MyDefinition), or the definition name (MyDefinition). The APIGW adds properties for added flexibility.
    • If you used definition: "request" in v5c, then in APIGW use validate-against: "body-param"
    • If you used definition: "response" in v5c, then in APIGW use  validate-against: "response-param"
    • If a definition was specified for "definition" in v5c (example #/definitions/error), then the comparable capability for the APIGW is to specify validate-against: "definition" and definition: "#/definitions/error"
    • The validate policy in the APIGW did not support the "definition name only" case (e.g. MyDefinition) in 2018.4.1.0, but it was added in 2018.4.1.1.

  • JWT Validate Policy

    • The output claims it is an object instead of a string

  • CORS support

    • APIGW will be compliant with the latest CORS spec at https://fetch.spec.whatwg.org/#cors-protocol
    • CORS support should not be enabled on both the GW and the backend.  v5c makes an attempt to handle the situation where CORS is enabled on the backend and on the gateway but is incomplete; for APIGW the user should have CORS support EITHER on the backend OR on the gateway, but not both (if user has CORS support enabled on both, gateway support may supercede backend support, but results are not guaranteed -- this is strongly not recommended).
    • In v5c, CORS support is not fully compliant with the current spec in some details.  Some examples follow.
      • v5c returns the header "Access-Control-Allow-Credentials: false", whereas APIGW will either return this header with a value of "true", or the header will not be present
      • v5c returns header "Access-Control-Expose-Headers" in response to a CORS preflight request, whereas APIGW will only return this header when responding to a simple CORS request, never in response to a preflight request.
      • APIGW returns CORS headers even when the request header "Origin" is empty or not present.  In APIGW only requests with a non-empty value of the "Origin" header will be considered CORS requests (note that the literal string "null" is a valid value for "Origin").

  • ?wsdl Support

    • Supported added in APIGW with 2018.4.1.1.

  • Vanity Endpoint URLs Support

    • Not supported in APIGW.

  • Extended Latency Logs

    • The same Extended Latency Log message is still used.
    • Two-letter/three-letter codes have been added that naturally reflect the native APIGW step they are timing.

2018.4.1.0 OAuth Details

The items below indicate adjustments in OAuth functionality between the DataPower Gateway (v5 compatible aka v5c) and the DataPower API Gateway in 2018.4.1.0 LTS. This information will help customers who are porting from V5c gateway to API Gateway in 2018.4.1.0 release.

  • Adjustments to the policy

    • Native OAuth providers now contain their own personal token secret instead of relying on the same secret provided by the gateway service.
    • Token management now saves the tokens in a whitelist so a distributed cache will be required to validate tokens on a different appliance from where the token was generated.
    • A native OAuth providers is capable of supporting both confidential and public clients. The native OAuth provider will default to a confidential client when a client secret is provided with a client ID. 
    • The client credentials for a request will be extracted from the query or form data before checking if a basic authorization header was provided. If an APIKey requirement is configured then the credentials obtained from the requirement will override any other client credentials provided.
    • The OAuth context variables have changed to account for the enhancement provided by moving the OAuth processing to the assembly.
    • Changes to the error messages and descriptions have been made to provide more useful and secure responses. To obtain more details about a response enable debug response headers and consult the debug logs.

  • Enhancements of the policy

    • OAuth token generation has been moved to the assembly which allows for advanced customization. The OAuth context can be modified to adjust the token(s) being generated.
    • Default scopes can be configured to enforce a scope on requests which don't provide a scope parameter.
    • The Advanced scope validation endpoint is configurable on the OAuth provider with the added ability to allow the consumer API to override the OAuth provider's endpoint.
    • Third party OAuth providers which require credentials can configure their credentials in the OAuth provider or indicate what header to provide the credentials.
    • Proof key for code exchange is now supported in native OAuth providers.

APIGW Assembly Coding Best Practices

  • Gateway Script (GWS)

    • Require statements are expensive. 
Required packages that are not actually being referenced should be removed.
    • Entry costs are high for GWS. 
Sequential GwS within an assembly should be merged into a single GwS action.
    • When coding for APIGW, access transaction variables via the context object rather than from the v5c apim object. 
Avoid require apim in GWS.

  • Map

    • Variable resolution within map is expensive
. In the map’s properties, disable “Resolve API Connect variable references”
    • Map is extremely versatile satisfying wide range of requirements
. For simple maps, consider coding in GWS (for JSON or XML data) or XSLT (for XML data) which can improve performance when implementing narrowly defined requirements.

  • Switch

    • Avoid extra casts, switch cases are always boolean regardless
, so you can avoid type casting as in: ($boolean(statusCode = '200'))

  • Parse

    • Remove unneeded parse policies. Parse should be used either because another policies requires parsed data as input (such as validate, xslt, json2xml, etc.), or because you want to enforce Threat Protection settings or calculate Parse Metrics.

Clone this wiki locally