Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Charm is in active/idle status even if it fails to get a certificate due to a 403 forbidden on the httprequest endpoint #154

Open
mthaddon opened this issue May 9, 2024 · 3 comments
Open

Charm is in active/idle status even if it fails to get a certificate due to a 403 forbidden on the httprequest endpoint #154

mthaddon opened this issue May 9, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@mthaddon
Copy link
Contributor

mthaddon commented May 9, 2024

Bug Description

If you deploy the charm and give it credentials and an endpoint that ends up with a 403, the charm is still reporting active/idle status.

To Reproduce

  1. Deploy the charm and give it credentials to connect to an endpoint that results in a 403
  2. See that it's in active/idle status with no indication of an error. Looking at juju debug-log is required

Environment

Running on Charmed K8s on top of Openstack. Using Juju 3.1.8.

Relevant log output

unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123: Exited with code 1. Stderr:
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] [cos-ps6.is-devops.canonical.com] acme: Obtaining bundled SA
N certificate given a CSR
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] [cos-ps6.is-devops.canonical.com] AuthURL: https://acme-v02.
api.letsencrypt.org/acme/authz-v3/348531099797
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] [cos-ps6.is-devops.canonical.com] acme: Could not find solve
r for: tls-alpn-01
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] [cos-ps6.is-devops.canonical.com] acme: Could not find solve
r for: http-01
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] [cos-ps6.is-devops.canonical.com] acme: use dns-01 solver
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] [cos-ps6.is-devops.canonical.com] acme: Preparing to solve D
NS-01
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] [cos-ps6.is-devops.canonical.com] acme: Cleaning DNS-01 chal
lenge
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [WARN] [cos-ps6.is-devops.canonical.com] acme: cleaning up failed: 
httpreq: unexpected status code: [status code: 403] body: <html>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <head><title>403 Forbidden</title></head>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <body>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <center><h1>403 Forbidden</h1></center>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <hr><center>nginx</center>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     </body>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     </html>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:54 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme
/authz-v3/348531099797
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     2024/05/09 07:49:55 Could not obtain certificates:
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:      error: one or more domains had a problem:
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     [cos-ps6.is-devops.canonical.com] [cos-ps6.is-devops.canonical.com] acme: error presenting token: httpreq: unexpected status code: [status code: 403] body: <html>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <head><title>403 Forbidden</title></head>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <body>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <center><h1>403 Forbidden</h1></center>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     <hr><center>nginx</center>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     </body>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123:     </html>
unit-httprequest-lego-k8s-0: 07:49:55 ERROR unit.httprequest-lego-k8s/0.juju-log certificates:123: Failed to execute lego command

Additional context

No response
@mthaddon mthaddon added the bug Something isn't working label May 9, 2024
@gruyaume
Copy link
Contributor

This issue is similar to #123 and #144 .

Httprequest can serve multiple TLS requirers. If it serves 2 requirers, and for the first one it returns 403, and for the second it returns 200, what should the status be?

Also, if there's a network error that prevents the endpoint from being reached for a moment, should the charm status be Blocked? Should the charm make curl calls every x min, to the provided endpoint to make sure it is continuously reachable? I am tempted to say no.

I understand the concern regarding the lack of feedback when requests don't work as expected, however I'm not convinced we should change the charm status.

@gruyaume gruyaume added enhancement New feature or request and removed bug Something isn't working labels May 10, 2024
@mthaddon
Copy link
Contributor Author

I think if it serves 2 requirers and one returns 403 and the other returns 200 the status should reflect that one has failed.

If there's a network error that prevents the endpoint from being reached for a while charms typically retry up to a fixed number of times and then either go into error state or blocked state.

I'm not sure why you're reluctant to change charm status. How else is the user supposed to know that there's been a problem getting a certificate?

@gruyaume
Copy link
Contributor

I think if it serves 2 requirers and one returns 403 and the other returns 200 the status should reflect that one has failed.

I am reluctant to change the charm status because it would require the charm to store a state for each of its relation. What we can do instead is reflect in the status message the number of certificate requests fulfilled. Ex.

Status     Message
Active     "1/3 certificate requests fulfilled"

If there's a network error that prevents the endpoint from being reached for a while charms typically retry

I agree that we should retry if it fails, I'm not sure whether that's the case right now. If it's not, we should do it.

and then either go into error state or blocked state.

I'm not sure about this, if the charm serves 99 requirers and everything works fine for 98 of them, but one of the requirers inserted a badly formatted common name, let's encrypt would return an error. Should httprequest go from active to blocked? My opinion is no it should not, the charm is functioning correctly. I agree we should do our best to reflect the information to the user. We do so with logs and we can improve the status message in the manner described above if it's useful.

cc. @ghislainbourgeois

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mthaddon @gruyaume