New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConfigMap booster - RoleBinding is not created properly before the tests execution #835

Closed
spisiakm opened this Issue Oct 26, 2017 · 21 comments

Comments

Projects
None yet
8 participants
@spisiakm
Contributor

spisiakm commented Oct 26, 2017

Issue Overview

This question is related to Vert.x ConfigMap booster. When I attempted to move the booster tests to Arquillian Cube, I encountered this issue. Before the tests execution, I want to deploy a configmap and a rolebinding (resource.roles.yaml file in .openshiftio folder in the booster repo) so that the app can access the configmap. Configmap is deployed fine, but the role binding is misconfigured - it's missing an entry in subjects and an entry in userNames (even though the entry in subjects list is specified in resource.roles.yaml and userNames should be created automatically) - see

bad-rolebinding

When I create the role binding manually using oc create -f resource.roles.yaml, it is created properly without any part missing, see

good-rolebinding

Expected Behaviour

A role binding is created properly before the tests are executed, with an entry in subjects and userNames present.

Current Behaviour

A role binding is missing the entry in subjects and userNames, which causes the app not being able to access the configmap - getting access denied, therefore causing the tests to fail.

Steps To Reproduce
  1. Clone the booster from https://github.com/spisiakm/vertx-configmap-booster-redhat
  2. Switch to arquillian-cube branch
  3. Run mvn clean verify -Popenshift,openshift-it when logged in some OpenShift instance, for example Minishift
  4. Check the pod logs, you should see something like io.vertx.core.impl.NoStackTraceThrowable: Access denied to configmap or secret in namespace booster: app-config
  5. Additionaly, when waiting for the tests to finish, you can do oc get rolebinding/role-view-default -o yaml to see the misconfigured role binding
Additional Information

The tests will take fairly long time to finish (fail) because of the await statements in the tests - you can change their timeout to a smaller number or just simply press Ctrl+c when you see enough.

Also both configmap and role binding are specified as env.dependencies in arquillian.xml

<details>
 <summary>$oc version</summary>
 oc v3.6.0
 kubernetes v1.6.1+5115d708d7
 features: Basic-Auth GSSAPI Kerberos SPNEGO

 Server https://192.168.42.214:8443
 openshift v3.6.0+c4dd4cf
 kubernetes v1.6.1+5115d708d7
</details>
@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Oct 27, 2017

Member

I will need to check it out deeper how it works but I think this is one of the use case that is supported in arq-ce and not in cube, so it is something to be ported. But we will need to check it out. Thakns.

Member

lordofthejars commented Oct 27, 2017

I will need to check it out deeper how it works but I think this is one of the use case that is supported in arq-ce and not in cube, so it is something to be ported. But we will need to check it out. Thakns.

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Oct 31, 2017

Member

Probably this issue can be fixed together with #824

Member

lordofthejars commented Oct 31, 2017

Probably this issue can be fixed together with #824

@luksa

This comment has been minimized.

Show comment
Hide comment
@luksa

luksa Nov 3, 2017

I've been looking at this today and have found the root cause. It's in the fabric8 k8s-client: it sent the RoleBinding with an empty groupNames and userNames fields (they're not specified in the YAML, but the F8 client adds them). The documentation says this:

$ oc explain rolebindings.subjects
     Subjects hold object references to authorize with this rule. This field is
     **ignored if UserNames or GroupNames** are specified...

Coincidentally, I fixed this a few days ago when fixing a different bug: fabric8io/kubernetes-client#895

luksa commented Nov 3, 2017

I've been looking at this today and have found the root cause. It's in the fabric8 k8s-client: it sent the RoleBinding with an empty groupNames and userNames fields (they're not specified in the YAML, but the F8 client adds them). The documentation says this:

$ oc explain rolebindings.subjects
     Subjects hold object references to authorize with this rule. This field is
     **ignored if UserNames or GroupNames** are specified...

Coincidentally, I fixed this a few days ago when fixing a different bug: fabric8io/kubernetes-client#895

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Nov 3, 2017

Member

Great, just one thing, can you comment this issue when the new client with this bug is fixed and released so we can update our dependencies as well?

Thank you very much.

Member

lordofthejars commented Nov 3, 2017

Great, just one thing, can you comment this issue when the new client with this bug is fixed and released so we can update our dependencies as well?

Thank you very much.

@Ladicek

This comment has been minimized.

Show comment
Hide comment
@Ladicek

Ladicek Nov 3, 2017

Contributor

Awesome! This actually prevents the Vert.x and Spring Boot ConfigMap boosters to have their tests migrated to Arquillian Cube, so it's great to see the progress (even if coincidental :-) ).

Contributor

Ladicek commented Nov 3, 2017

Awesome! This actually prevents the Vert.x and Spring Boot ConfigMap boosters to have their tests migrated to Arquillian Cube, so it's great to see the progress (even if coincidental :-) ).

@spisiakm

This comment has been minimized.

Show comment
Hide comment
@spisiakm

spisiakm Nov 6, 2017

Contributor

This should be fixed in kubernetes-client v3.0.3 (released 3 days ago). I haven't tried that yet though.

Contributor

spisiakm commented Nov 6, 2017

This should be fixed in kubernetes-client v3.0.3 (released 3 days ago). I haven't tried that yet though.

@Ladicek

This comment has been minimized.

Show comment
Hide comment
@Ladicek

Ladicek Nov 6, 2017

Contributor

I'm pretty sure F8 K8s client 3.0 has breaking changes compared to 2.6, so not sure if/when Arquillian Cube would like to deal with those.

@luksa is there a possibility of having a 2.6.x release with this fix? :-)

Contributor

Ladicek commented Nov 6, 2017

I'm pretty sure F8 K8s client 3.0 has breaking changes compared to 2.6, so not sure if/when Arquillian Cube would like to deal with those.

@luksa is there a possibility of having a 2.6.x release with this fix? :-)

@bartoszmajsak

This comment has been minimized.

Show comment
Hide comment
@bartoszmajsak

bartoszmajsak Nov 6, 2017

Member

Do we have a reproducer for that? If so we can always try if 3.0.3 is fine to bump to. cc @lordofthejars

Member

bartoszmajsak commented Nov 6, 2017

Do we have a reproducer for that? If so we can always try if 3.0.3 is fine to bump to. cc @lordofthejars

@luksa

This comment has been minimized.

Show comment
Hide comment
@luksa

luksa Nov 6, 2017

@Ladicek I'm not on the fabric8 team, so you'll have to talk to @iocanel to see if they can release 2.6.x

luksa commented Nov 6, 2017

@Ladicek I'm not on the fabric8 team, so you'll have to talk to @iocanel to see if they can release 2.6.x

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Nov 7, 2017

Member

Just one note updating from 2.X to 3.X is not an easy task because of this shading thing we are using. But of course I am not saying is not possible but it will take some time to do it, it isnot just a matter of updating a property.

Member

lordofthejars commented Nov 7, 2017

Just one note updating from 2.X to 3.X is not an easy task because of this shading thing we are using. But of course I am not saying is not possible but it will take some time to do it, it isnot just a matter of updating a property.

@Ladicek

This comment has been minimized.

Show comment
Hide comment
@Ladicek

Ladicek Nov 7, 2017

Contributor

Yea that's why I'm thinking if we could get a new 2.6 release with this fix. @iocanel WDYT?

EDIT: it's actually pretty important for us (RHOAR in general and RHOAR QE in particular), as it blocks the Spring Boot and Vert.x ConfigMap boosters tests to be migrated to Arquillian Cube.

Contributor

Ladicek commented Nov 7, 2017

Yea that's why I'm thinking if we could get a new 2.6 release with this fix. @iocanel WDYT?

EDIT: it's actually pretty important for us (RHOAR in general and RHOAR QE in particular), as it blocks the Spring Boot and Vert.x ConfigMap boosters tests to be migrated to Arquillian Cube.

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Nov 7, 2017

Member

@Ladicek if could be in 2.6 this would be great, if not, I am not saying it is not possible :) but it will take more time so you can use it.

Member

lordofthejars commented Nov 7, 2017

@Ladicek if could be in 2.6 this would be great, if not, I am not saying it is not possible :) but it will take more time so you can use it.

@iocanel

This comment has been minimized.

Show comment
Hide comment
@iocanel

iocanel Nov 7, 2017

Contributor

We rarely (if ever) cut releases for previous major versions (we never had to so far). I am not sure if our release pipelines ever support it. This applies to upstream. From time to time I do see product releases (from older major versions) but I am not sure who drives them. So I'll have to check.

Contributor

iocanel commented Nov 7, 2017

We rarely (if ever) cut releases for previous major versions (we never had to so far). I am not sure if our release pipelines ever support it. This applies to upstream. From time to time I do see product releases (from older major versions) but I am not sure who drives them. So I'll have to check.

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Nov 9, 2017

Member

@iocanel so maybe the best thing is update Arquillian Cube right? The only thing we need to take care is of changing the imports to V3 right?

Member

lordofthejars commented Nov 9, 2017

@iocanel so maybe the best thing is update Arquillian Cube right? The only thing we need to take care is of changing the imports to V3 right?

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Nov 21, 2017

Member

@Ladicek @alesj In this PR #856 we have updated to Kubernetes-client 3.1.0 and it is on master. Can you try this if now it works and if so then I will link the PR to this issue and close it.

Thank you very much.

Member

lordofthejars commented Nov 21, 2017

@Ladicek @alesj In this PR #856 we have updated to Kubernetes-client 3.1.0 and it is on master. Can you try this if now it works and if so then I will link the PR to this issue and close it.

Thank you very much.

@spisiakm

This comment has been minimized.

Show comment
Hide comment
@spisiakm

spisiakm Nov 21, 2017

Contributor

Hi @lordofthejars , I have just tried this with current master (1.9.3-SNAPSHOT), but I'm getting this exception:
arq_cube_error
Could it be that Arquillian Cube just ignores the wait.timeout (set to 300000ms by default) ? Because, as you can see in the picture, I'm getting this error just after ~2.3 secs, which is definitely not 300 secs.

Contributor

spisiakm commented Nov 21, 2017

Hi @lordofthejars , I have just tried this with current master (1.9.3-SNAPSHOT), but I'm getting this exception:
arq_cube_error
Could it be that Arquillian Cube just ignores the wait.timeout (set to 300000ms by default) ? Because, as you can see in the picture, I'm getting this error just after ~2.3 secs, which is definitely not 300 secs.

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Nov 21, 2017

Member

In theory not, in fact we have not changed anything regarding this. I remember that this happened to me as well, and I thought the same as you, but in that case it was an error in my project, but now I cannot remember exactly the problem.

Can you share the project? If I have time I'll try to see what's happening or if I see something, also I suggest you debug where the exception is thrown to try to identify if maybe there is a bug/missconfigured in the new Kuberentes-client.

Member

lordofthejars commented Nov 21, 2017

In theory not, in fact we have not changed anything regarding this. I remember that this happened to me as well, and I thought the same as you, but in that case it was an error in my project, but now I cannot remember exactly the problem.

Can you share the project? If I have time I'll try to see what's happening or if I see something, also I suggest you debug where the exception is thrown to try to identify if maybe there is a bug/missconfigured in the new Kuberentes-client.

@spisiakm

This comment has been minimized.

Show comment
Hide comment
@spisiakm

spisiakm Nov 21, 2017

Contributor

@lordofthejars sure, just clone https://github.com/spisiakm/vertx-configmap-booster , switch to branch arquillian-cube and run mvn clean verify -Popenshift,openshift-it provided you have Arquillian Cube 1.9.3-SNAPSHOT installed and configured to be used.

Contributor

spisiakm commented Nov 21, 2017

@lordofthejars sure, just clone https://github.com/spisiakm/vertx-configmap-booster , switch to branch arquillian-cube and run mvn clean verify -Popenshift,openshift-it provided you have Arquillian Cube 1.9.3-SNAPSHOT installed and configured to be used.

@dipak-pawar dipak-pawar self-assigned this Nov 21, 2017

@alesj

This comment has been minimized.

Show comment
Hide comment
@alesj

alesj Nov 21, 2017

Collaborator

It works for me:
[INFO] Spring Boot - ConfigMap Booster - Tests ............ SUCCESS [01:28 min]

Collaborator

alesj commented Nov 21, 2017

It works for me:
[INFO] Spring Boot - ConfigMap Booster - Tests ............ SUCCESS [01:28 min]

@spisiakm

This comment has been minimized.

Show comment
Hide comment
@spisiakm

spisiakm Nov 21, 2017

Contributor

Hmm, you're right @lordofthejars , it looks like it is an issue with maven dependencies on my side, possibly some conflicts. Vert.x ConfigMap booster works fine with Vert.x v3.5.0, but doesn't work with v3.4.2. I'll try to investigate further, but this is another issue. So I guess I can say upgrading kubernetes-client in #856 fixes this issue. Thank you kindly!

Contributor

spisiakm commented Nov 21, 2017

Hmm, you're right @lordofthejars , it looks like it is an issue with maven dependencies on my side, possibly some conflicts. Vert.x ConfigMap booster works fine with Vert.x v3.5.0, but doesn't work with v3.4.2. I'll try to investigate further, but this is another issue. So I guess I can say upgrading kubernetes-client in #856 fixes this issue. Thank you kindly!

@lordofthejars

This comment has been minimized.

Show comment
Hide comment
@lordofthejars

lordofthejars Nov 21, 2017

Member

@spisiakm Thank you to you for rising this. Then looks I am going to close the issue but if you find anything else, please just open an ew issue and we will implement ASAP.

Member

lordofthejars commented Nov 21, 2017

@spisiakm Thank you to you for rising this. Then looks I am going to close the issue but if you find anything else, please just open an ew issue and we will implement ASAP.

@lordofthejars lordofthejars added this to the 1.9.3 milestone Nov 21, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment