THREESCALE-10518 - Operator crash - OpenAPI CR doesn't create correct backend...

[valerymo]

Closing current PR, as commits looks not clear after squash / rebase process (probably mistake was done)
Please see new PR created with changes of this PR #915
Could you please approve new PR

WHAT

Jira: https://issues.redhat.com/browse/THREESCALE-10518
Operator crash - OpenAPI CR doesn't create correct backend usage section in Product CR

Validation - Preparation

Prepare cluster (could be OSD)

Install RHSSO

Create project rhsso-test

1. Install RHSSO from OperatorHub into project rhsso-test
2. Create Keycloak Instance of RHSSO
3. Open RHSSO portal
4. Create a realm for the petstore product
   In RH User SSO web console:

• Click Add realm. Add realm page will be opened
• Set name, as petstore
• Click Create. Petstore realm page will be opened. No changes required.

5. Create a client for 3scale

• In RH User SSO web console:

  • Choose Petstore realm (in top-left corner)
  • Select Configure -> Client
  • Click Create
  • Set Client ID: 3scale-zync
  • Click Save. Clients->3scale-zync page will be opened

• Client Settings:

3scale-zync Client setting will be as in the table below

Parameter	Value
Name	3scale-zync
Client Protocol	openid-connect
Access Type	confidential
Standard Flow Enabled	Off
Direct Access Grants Enabled	Off
Service Accounts Enabled	On

• Click Save
• Provide Realm-Management - manage-clients:
  • Enter Service Account Roles tab -> Client Roles
    • Choose realm-management -> manage-clients
    • Click Add selected

Install 3scale

1. Install and run 3scale operator

cd 3scale-operator
make install
oc new-project 3scale-test
make download
make run

2. Apply s3 secret

• Secret example (s3-creds-secret.yaml):

kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: QU12345
  AWS_SECRET_ACCESS_KEY: aU12345=
  AWS_BUCKET: dm12345=
  AWS_REGION: ZX1234==
type: Opaque

oc apply -f s3-creds-secret.yaml

3. Apply Apimanager CR

• Apimanager CR example (apimanagerCR.yaml),
  please place your wildcardDomain
• apimanagerCR.yaml:

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
    name: example-apimanager
    namespace: 3scale-test
spec:
    wildcardDomain: apps.vmogilev01.xxx.s1.devshift.org

oc apply -f apimanagerCR.yaml

6. Apply oidc issuer client secret
   This is the secret that contains URL for issuerEndpoint.
   The secret is referenced in OpenApi CR - field issuerEndpointRef.

• issuerEndpoint meed to be encryupted; below - just for sample
• oidc-issuer-client-secret.yaml:

kind: Secret
apiVersion: v1
metadata:
  name: oidc-issuer-client-secret
  namespace: 3scale-test
data:
  issuerEndpoint: https://3scale-zync:some-secret@keycloak-rhsso-test.apps.xxxxx.xxxx.xx.xx.org/auth/realms/petstore
type: Opaque

oc apply -f oidc-issuer-client-secret.yaml

Openapi Secret

Apply Secret that contains OpenApi spec

• securitySchemes type: oauth2

NOTE Two OAS securitySchemes will be tested - oauth2 and openidConnect. See more details in Validation section.

• my-openapi-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: my-openapi-secret
type: Opaque
stringData:
  myopenapi.yaml: |
    ---
    components:
      schemas:
        Error:
          properties:
            code:
              format: int32
              type: integer
            message:
              type: string
          required:
          - code
          - message
          type: object
        NewPet:
          properties:
            name:
              type: string
            tag:
              type: string
          required:
          - name
          type: object
        Pet:
          allOf:
          - $ref: '#/components/schemas/NewPet'
          - properties:
              id:
                format: int64
                type: integer
            required:
            - id
            type: object
      securitySchemes:
        myOauth:
          description: This API uses OAuth 2 with the implicit grant flow. [More info](https://api.example.com/docs/auth)
          flows:
            password:
              scopes:
                read_pets: read your pets
                write_pets: modify pets in your account
              tokenUrl: https://api.example.com/oauth2/token
          type: oauth2
    info:
      contact:
        email: apiteam@swagger.io
        name: Swagger API Team
        url: http://swagger.io
      description: A sample API that uses a petstore as an example to demonstrate features
        in the OpenAPI 3.0 specification
      license:
        name: Apache 2.0
        url: https://www.apache.org/licenses/LICENSE-2.0.html
      termsOfService: http://swagger.io/terms/
      title: Swagger Petstore M
      version: 1.0.0
    openapi: 3.0.0
    paths:
      /pets:
        get:
          description: 'Returns all pets from the system that the user has access to

            Nam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem
            sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio
            lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar
            ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer
            at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie
            imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis.
            Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend
            fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat
            id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum
            libero. Morbi euismod sagittis libero sed lacinia.


            Sed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum
            ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis
            elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque
            posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend
            nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce
            malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique,
            sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan
            tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend
            lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien.

            '
          operationId: findPets
          parameters:
          - description: tags to filter by
            in: query
            name: tags
            required: false
            schema:
              items:
                type: string
              type: array
            style: form
          - description: maximum number of results to return
            in: query
            name: limit
            required: false
            schema:
              format: int32
              type: integer
          responses:
            '200':
              content:
                application/json:
                  schema:
                    items:
                      $ref: '#/components/schemas/Pet'
                    type: array
              description: pet response
            default:
              content:
                application/json:
                  schema:
                    $ref: '#/components/schemas/Error'
              description: unexpected error
          security:
          - myOauth:
            - read
            - write
        post:
          description: Creates a new pet in the store. Duplicates are allowed
          operationId: addPet
          requestBody:
            content:
              application/json:
                schema:
                  $ref: '#/components/schemas/NewPet'
            description: Pet to add to the store
            required: true
          responses:
            '200':
              content:
                application/json:
                  schema:
                    $ref: '#/components/schemas/Pet'
              description: pet response
            default:
              content:
                application/json:
                  schema:
                    $ref: '#/components/schemas/Error'
              description: unexpected error
          security:
          - myOauth:
            - read
            - write
      /pets/{id}:
        delete:
          description: deletes a single pet based on the ID supplied
          operationId: deletePet
          parameters:
          - description: ID of pet to delete
            in: path
            name: id
            required: true
            schema:
              format: int64
              type: integer
          responses:
            '204':
              description: pet deleted
            default:
              content:
                application/json:
                  schema:
                    $ref: '#/components/schemas/Error'
              description: unexpected error
          security:
          - myOauth:
            - read
            - write
        get:
          description: Returns a user based on a single ID, if the user does not have
            access to the pet
          operationId: find pet by id
          parameters:
          - description: ID of pet to fetch
            in: path
            name: id
            required: true
            schema:
              format: int64
              type: integer
          responses:
            '200':
              content:
                application/json:
                  schema:
                    $ref: '#/components/schemas/Pet'
              description: pet response
            default:
              content:
                application/json:
                  schema:
                    $ref: '#/components/schemas/Error'
              description: unexpected error
          security:
          - myOauth:
            - read
            - write
    security:
    - myOauth:
      - read
      - write
    servers:
    - url: http://petstore.swagger.io/v1

oc apply -f my-openapi-secret.yaml

Validation - Tests

Test cases

Following cases will be tested
OpenApiCR is OIDC. OAS securitySheme type: oauth2

• Test1. Invalid OpenApi CR. OIDC Issuer Endpoint definition is missing in CR.

  • Expected result: Product Not created. Status Error in Openapi CR:

  OIDC issuer endpoint definition is missing in CR - No IssuerEndpoint nor IssuerEndpointRef found in OIDC spec in CR, one of them must be set.
  

• Test2. Valid OpenApi CR (oidc), Valid OAS (oauth2)

  • Expected result: Product created. Product's OIDC authentication flows settings done according to OAS flows. For more info please see PR THREESCALE-10523 - oidc in OpenAPI CR ignores some attributes for Product CR #909 ,

    3scale-operator/doc/openapi-user-guide.md

    Lines 191 to 196 in 185e1ae

    	- **If OpenAPI CR spec is OIDC but securitySchemes type in OAS is oauth2** then CR OIDC Authentication Flows parameters will be ignored,
    	and Product OIDC Authentication Flows will be set to match oauth2 flows that defined in OAS, as following
    	- StandardFlowEnabled = true if oauth2 AuthorizationCode is defined
    	- ImplicitFlowEnabled = true if oauth2 Implicit is defined
    	- DirectAccessGrantsEnabled = true if oauth2 Password is defined
    	- ServiceAccountsEnabled = true if oauth2 ClientCredentials is defined

• Test3. Invalid OAS - missing flows definition in OAS securitySheme

  • Expected result: Product Not created. Status Error in Openapi CR:

  Status:
  Conditions:
      Last Transition Time:  2024-01-09T16:39:30Z
      Message:               spec.openapiRef.secretRef: Invalid value: v1.ObjectReference{Kind:"", Namespace:"3scale-test", Name:"openapi-secret-oauth2-prod3", UID:"", APIVersion:"", ResourceVersion:"", FieldPath:""}: invalid components: security scheme of type "oauth2" should have 'flows'
  

OpenApiCR is OIDC. OAS securitySheme type: openIdConnect

• Test4. Valid OpenApi CR (OIDC), Valid OAS (OIDC)
  • Expected result: Product created. Product's OIDC authentication flows settings done according to flows defined in openapi CR.

OIDC not defined in OpenApiCR spec. OAS securitySheme type: one of oauth2 or openIdConnect

• Test5. OIDC not defined in OpenApiCR spec. OAS securitySheme type: one of oauth2 or openIdConnect
  • Expected result: Product Not created. Status Error in Openapi CR:

  Status:
  Conditions:
  Last Transition Time:  2024-01-10T07:44:41Z
  Message:               spec.openapiRef: Invalid value: v1beta1.OpenAPIRefSpec{SecretRef:(*v1.ObjectReference)(0xc000208700), URL:(*string)(nil)}: Missing OIDC definitions in CR. The referenced OpenAPI spec's sec scheme is openIdConnect or oauth2, the spec.oidc must not be nil or empty
  

OpenApiCR is OIDC. OAS securitySheme not defined

• Test6. Valid OpenApi CR (OIDC), Not secured OAS
• Expected result: Product created with UserKey authentication. Warning event in Openapi CR:

Events:
  Type     Reason                                     Age                From     Message
  ----     ------                                     ----               ----     -------
  Warning  No security requirements are found in OAS  9s (x12 over 17s)  OpenAPI  OIDC definitions in CR will be ignored, as no security requirements are found. Default to UserKey authentication

Tests Examples

Test1. Invalid OpenApi CR. OIDC Issuer Endpoint definition is missing in CR

• Expected result: Product Not created. Status Error in Openapi CR:

OIDC issuer endpoint definition is missing in CR - No IssuerEndpoint nor IssuerEndpointRef found in OIDC spec in CR, one of them must be set.

• Apply OpenAPI CR
• openapi-example.yaml:

apiVersion: capabilities.3scale.net/v1beta1
kind: OpenAPI
metadata:
  annotations:
    insecure_skip_verify: 'true'
  name: openapi-example
  namespace: 3scale-test
spec:
  oidc:
    issuerType: keycloak
  openapiRef:
    secretRef:
      name: my-openapi-secret
      namespace: 3scale-test
  prefixMatching: true

oc apply -f openapi-example.yaml

• Error in openapi CR:

$ oc describe openapi
Name:         openapi-example
Namespace:    3scale-test
Labels:       <none>
Annotations:  insecure_skip_verify: true
API Version:  capabilities.3scale.net/v1beta1
Kind:         OpenAPI
Metadata:
......
......
Status:
  Conditions:
    Last Transition Time:  2024-01-10T06:04:28Z
    Message:               spec.openapiRef: Invalid value: v1beta1.OpenAPIRefSpec{SecretRef:(*v1.ObjectReference)(0xc0004f0230), URL:(*string)(nil)}: OIDC issuer endpoint definition is missing in CR - No IssuerEndpoint nor IssuerEndpointRef found in OIDC spec in CR, one of them must be set.
    Status:                True
    Type:                  Failed
    Last Transition Time:  2024-01-10T06:04:28Z
    Message:               spec.openapiRef: Invalid value: v1beta1.OpenAPIRefSpec{SecretRef:(*v1.ObjectReference)(0xc0004f0230), URL:(*string)(nil)}: OIDC issuer endpoint definition is missing in CR - No IssuerEndpoint nor IssuerEndpointRef found in OIDC spec in CR, one of them must be set.
    Status:                True
    Type:                  Invalid
    Last Transition Time:  2024-01-10T06:04:28Z
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason                Age                     From     Message
  ----     ------                ----                    ----     -------
  Warning  Invalid OpenAPI Spec  21m                     OpenAPI  spec.openapiRef: Invalid value: v1beta1.OpenAPIRefSpec{SecretRef:(*v1.ObjectReference)(0xc0003ef030), URL:(*string)(nil)}: OIDC issuer endpoint definition is missing in CR - No IssuerEndpoint nor IssuerEndpointRef found in OIDC spec in CR; please set it to fix the problem.
  Warning  Invalid OpenAPI Spec  21m                     OpenAPI  spec.openapiRef: Invalid value: v1beta1.OpenAPIRefSpec{SecretRef:(*v1.ObjectReference)(0xc0008b8000), URL:(*string)(nil)}: OIDC issuer endpoint definition is missing in CR - No IssuerEndpoint nor IssuerEndpointRef found in OIDC spec in CR; please set it to fix the problem.

Test2 - Valid OpenApi CR (oidc), Valid OAS (oauth2)

• Edit existing OpenAPI CR: 1) change name 2) add issuerEndpointRef as in example below

apiVersion: capabilities.3scale.net/v1beta1
kind: OpenAPI
metadata:
  annotations:
    insecure_skip_verify: 'true'
  name: openapi-example
  namespace: 3scale-test
spec:
  oidc:
    issuerType: keycloak
    issuerEndpointRef:
      name: oidc-issuer-client-secret
  openapiRef:
    secretRef:
      name: my-openapi-secret
      namespace: 3scale-test
  prefixMatching: true

- *Expected result*: Product created. Product's OIDC authentication flows settings done according to OAS flows. For more info please see PR https://github.com/3scale/3scale-operator/pull/909 , https://github.com/3scale/3scale-operator/blob/185e1ae26ffe824988eef75716a69d4388f8d90f/doc/openapi-user-guide.md?plain=1#L191-L196 

Notes for other tests execution

• Other tests can be executed by modifing OpenapiCR and OAS (that located in secret).
• for easy cases control - you can add suffix to name in OpenapiCR corresponding to test name, like openapi-example-2. Also add suffix to OAS titile, like title: Swagger Petstore 2

[valerymo]

@MStokluska , hope review comments addressed now. It was discussed in Team chat and googleDoc. Could you please review here. Thanks for your comments.
I also updated Validation section to make tests cases more clear. Hope it will help.

[MStokluska]

/lgtm

Looks good to me. Thanks Valery

[MStokluska]

Just one note, the Commit message should be <JIRA_NO> - having a <JIRA_NO>- might affect how the PR is consumed by our build system. Please update.

[austincunningham]

Extra hiphens added here . I know this was rasised before and the IDE is doing it. You can disable table formatting in the ide settings to stop this from happening

[austincunningham]

My bad after testing it doesn't work 🤦

[valerymo]

Updated. Thank you for review and help notes.

[valerymo]

Just one note, the Commit message should be <JIRA_NO> - having a <JIRA_NO>- might affect how the PR is consumed by our build system. Please update.

Thank you for review and help notes!

[openshift-ci]

@valerymo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-e2e	59963bd	link	true	/test test-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[codeclimate]

Code Climate has analyzed commit 893262b and detected 13 issues on this pull request.

Here's the issue category breakdown:

Category	Count
Complexity	2
Duplication	4
Style	7

View more on Code Climate.

[valerymo]

@austincunningham @MStokluska
Closing current PR, as commits looks not clear after squash / rebase process (probably mistake was done)
Please see new PR created with changes of this PR #915
Could you please approve new PR. Sorry for problems. Thanks