OAuth2 authentication is broken when using k8s with istio

[deepu105]

When creating a microservice stack with Oauth2 auth and deploying to istio on GKE, authentication to microservices ends up in an infinite loop. The gateway application and authentication for that works, but if you try to access an API provided by one of the microservice it will go on a loop. I was trying to get my standard e-commerce ms stack working with Keycloak and Okta and ended up here. Still trying to figure out whats going on. I don't see any authorization headers being passed around. But I don't see those headers for gateway APIs as well, for example the customer endpoint served by store. I'm not sure if its an Istio issue though, probably how auth is setup on JH app side.

I'll get back to debugging it after holidays, so in the meantime if anyone wants to give it a shot please go ahead

Sample JDL

/*
 * This is a microservice e-commerce store sample with Gateway and three microservice applications
 * This uses Istio for native kubernetes service discovery and oauth2 authentication
 * This also creates the required Kubernetes deployment manifests and Istio routes
 */
 
application {
  config {
    baseName store
    applicationType gateway
    packageName com.okta.developer.store
    serviceDiscoveryType no
    authenticationType oauth2
    prodDatabaseType postgresql
    cacheProvider hazelcast
    buildTool gradle
    clientFramework react
  }
  entities *
}


application {
  config {
    baseName product
    applicationType microservice
    packageName com.okta.developer.product
    serviceDiscoveryType no
    authenticationType oauth2
    prodDatabaseType postgresql
    cacheProvider hazelcast
    buildTool gradle
    serverPort 8081
  }
  entities Product, ProductCategory, ProductOrder, OrderItem
}

application {
  config {
    baseName invoice
    applicationType microservice
    packageName com.okta.developer.invoice
    serviceDiscoveryType no
    authenticationType oauth2
    prodDatabaseType postgresql
    buildTool gradle
    serverPort 8082
  }
  entities Invoice, Shipment
}

application {
  config {
    baseName notification
    applicationType microservice
    packageName com.okta.developer.notification
    serviceDiscoveryType no
    authenticationType oauth2
    databaseType mongodb
    cacheProvider no
    enableHibernateCache false
    buildTool gradle
    serverPort 8083
  }
  entities Notification
}

/**
 * Entities for Store Gateway
 */

// Customer for the store
entity Customer {
    firstName String required
    lastName String required
    gender Gender required
    email String required pattern(/^[^@\s]+@[^@\s]+\.[^@\s]+$/)
    phone String required
    addressLine1 String required
    addressLine2 String
    city String required
    country String required
}

enum Gender {
    MALE, FEMALE, OTHER
}

relationship OneToOne {
    Customer{user(login) required} to User
}

service Customer with serviceClass
paginate Customer with pagination


/**
 * Entities for product microservice
 */


// Product sold by the Online store 
entity Product {
    name String required
    description String
    price BigDecimal required min(0)
    itemSize Size required
    image ImageBlob
}

enum Size {
    S, M, L, XL, XXL
}

entity ProductCategory {
    name String required
    description String
}

entity ProductOrder {
    placedDate Instant required
    status OrderStatus required
    code String required
    invoiceId Long
    customer String required
}

enum OrderStatus {
    COMPLETED, PENDING, CANCELLED
}

entity OrderItem {
    quantity Integer required min(0)
    totalPrice BigDecimal required min(0)
    status OrderItemStatus required
}

enum OrderItemStatus {
    AVAILABLE, OUT_OF_STOCK, BACK_ORDER
}

relationship ManyToOne {
	OrderItem{product(name) required} to Product
}

relationship OneToMany {
   ProductOrder{orderItem} to OrderItem{order(code) required} ,
   ProductCategory{product} to Product{productCategory(name)}
}

service Product, ProductCategory, ProductOrder, OrderItem with serviceClass
paginate Product, ProductOrder, OrderItem with pagination
microservice Product, ProductOrder, ProductCategory, OrderItem with product


/**
 * Entities for Invoice microservice
 */


// Invoice for sales
entity Invoice {
    code String required
    date Instant required
    details String
    status InvoiceStatus required
    paymentMethod PaymentMethod required
    paymentDate Instant required
    paymentAmount BigDecimal required
}

enum InvoiceStatus {
    PAID, ISSUED, CANCELLED
}

entity Shipment {
    trackingCode String
    date Instant required
    details String
}

enum PaymentMethod {
    CREDIT_CARD, CASH_ON_DELIVERY, PAYPAL
}

relationship OneToMany {
    Invoice{shipment} to Shipment{invoice(code) required}
}

service Invoice, Shipment with serviceClass
paginate Invoice, Shipment with pagination
microservice Invoice, Shipment with invoice


/**
 * Entities for notification microservice
 */


entity Notification {
    date Instant required
    details String
    sentDate Instant required
    format NotificationType required
    userId Long required
    productId Long required
}

enum NotificationType {
    EMAIL, SMS, PARCEL
}

microservice Notification with notification

[mohammadsrahman]

I experienced the same issue today. I managed to get authorisation to work I think by editing the gateway virtual service. For some reason looks like calls are being made to /service/api/api so adding the extra api to the virtual service kind of stopped the authorisation issue but I am experiencing other issues where entities are not being returned.

[mshima]

I recall seen this and created this branch #17388, but it needs more testing.
The expired token is ignored at the gateway side, but not at the microservice side, creating this infinite loop.

[deepu105]

You are too fast Marcelo ❤️ I'll do some tests with your branch and let you know

…

On Thu, 23 Dec 2021, 12:00 am Marcelo Shima, ***@***.***> wrote: I recall seen this and created this branch #17388 <#17388>, but it needs more testing. The expired token is ignored at the gateway side, but not at the microservice side, creating this infinite loop. — Reply to this email directly, view it on GitHub <#17384 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIOKFZDD27KEDHOPPBJD7LUSJKAPANCNFSM5KS4ZMBQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[deepu105]

I tested your branch but the looping issue is still present at least with Istio + k8s

[mshima]

The PR was from a previous problem I found.
Not actively working on this right now.

[deepu105]

no worries

[deepu105]

for some more context, normal microservices with reactive gateway works fine with OAuth, its only with Istio there is issue.

[taoyonggang]

This bug isn't fixed at latest version.

[taoyonggang]

The right solution is removing the auth fuction from code .And then add OPA config with Istio.

Ref:
https://istio.io/latest/blog/2021/better-external-authz/
https://medium.com/google-cloud/external-authorization-server-with-istio-1159b21682bb

[deepu105]

Probably, but the gateway is able to login with OAuth and microservices hence should be able to auth using JWT so maybe configuring OPA for that could work without removing auth Thanks & Regards, Deepu

…

On Mon, Jan 24, 2022 at 7:45 AM taoyonggang ***@***.***> wrote: The right solution is removing the auth fuction from code .And then add OPA config with Istio. Ref: https://istio.io/latest/blog/2021/better-external-authz/ https://medium.com/google-cloud/external-authorization-server-with-istio-1159b21682bb — Reply to this email directly, view it on GitHub <#17384 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIOKF3BIZATGI6GGZI4VKTUXSY5LANCNFSM5KS4ZMBQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[DanielFran]

@taoyonggang Can you provide a PR?

[deepu105]

Hey @xetys could use some of your expertise here if you have time

[ohmex]

I wasted one full day before landing here !

[deepu105]

@ohmex do share what you learned. Maybe I missed something during my debugging