IAM Roles (IRSA) doesn't work with gremlin-console

[talmarco]

I'm trying to configure IRSA (IAM Roles) when using neptune-gremlin, but without success.
I have created an IAM policy, attached this policy to a role and this role is also attached to a Kubernetes service account by mentioning the role name as an annotation.

Service Account:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/role-name
  name: neptune-sa
  namespace: dev

Policy:

{
  "Effect": "Allow",
  "Action": "neptune-db:*",
  "Resource": "arn:aws:neptune-db:<our-region>:<account-id>:cluster-<cluster-id>/*"
}

I'm creating the pod using the following command:

kubectl run gremlin-console \
  --rm=true -i --tty --image <gremlin-console-image> --env="SERVICE_REGION=<our-region>"  \
  <endpoint-name>.<our-region>.neptune.amazonaws.com -n dev \
  --overrides='{ "spec": { "serviceAccountName": "neptune-sa" }  }'

The error I'm getting is:

ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler  - Could not process the response
io.netty.handler.codec.http.websocketx.WebSocketHandshakeException: Invalid handshake response getStatus: 403 Forbidden
        at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:274)
        at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:302)
        at org.apache.tinkerpop.gremlin.driver.handler.WebSocketClientHandler.channelRead0(WebSocketClientHandler.java:69)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
        at io.netty.handler.codec.http.websocketx.extensions.WebSocketClientExtensionHandler.channelRead(WebSocketClientExtensionHandler.java:127)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)                                                                                                                                                                                                                 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282)
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)                                                                                                                                                                                                                          at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run(Thread.java:748)

As a side note; this mechanism works for me on all other pods but Gremlin <> Neptune.

Thanks!

[krlawrence]

If you have IAM authentication enabled on the Neptune cluster, the call to Neptune has to be signed using SigV4. Are you signing that request?

[talmarco]

If you have IAM authentication enabled on the Neptune cluster, the call to Neptune has to be signed using SigV4. Are you signing that request?

IAM authentication is enabled. My hunch is gremlin-console doesn't support WebIdentityToken when signing the request.

[afreeland]

@talmarco seeing this as well, was curious if you had ever resolved it? I noticed this when trying to use the neptune-export utility from the neptune user guide documentation.

[krlawrence]

Hi @afreeland - can you say a bit more about the exact steps you are trying and the failure you are seeing?

[talmarco]

@talmarco seeing this as well, was curious if you had ever resolved it? I noticed this when trying to use the neptune-export utility from the neptune user guide documentation.

Nope, still have the same problem

[afreeland]

@krlawrence I am a bit embarrassed to say but I believe it was because I was missing the AWS_ACCESS_KEY_ID and the AWS_SECRET_ACCESS_KEY. I had the SERVICE_REGION defined but was going kind of out of band in our environment and wasn't getting the two env vars that were required.

For me I was using the neptune-export.jar and didn't have much control over the signature part but was seeing the exporter throw errors very similar to the above.

I did all the checks to make sure VPC was looking good and then was able to use curl 7.79.1 with the new --aws-sigv4 parameter to hit neptune cluster...and was like, okay...let me take some steps back lol.

I believe once I added those env vars life was good and connectivity seemed to work...so my use-case may be different than the original thread.

[Piroddi]

Hi @talmarco, experiencing the same. Were you ever able to resolve ?

[talmarco]

@Piroddi Unfortunately i didn't manage to resolve this