oidc-client-relation-broken hook failed

[jardon]

dex-auth charm goes into error state on an oidc-client-relation-broken hook failure. Will update the issue when I can determine exact steps to reproduce the issue.

2022-03-03 06:42:40 INFO juju-log Running legacy hooks/upgrade-charm.
2022-03-03 06:42:41 INFO juju.worker.caasoperator.uniter.dex-auth/5.operation runhook.go:152 ran "upgrade-charm" hook (via hook dispatching script: dispatch)
2022-03-03 06:42:41 INFO juju.worker.caasoperator.uniter.dex-auth/5 resolver.go:154 found queued "config-changed" hook
2022-03-03 06:42:43 ERROR juju-log oidc-client:2: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 1284, in _run
    result = run(args, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-dex-auth-4/relation-get', '-r', '2', '-', '', '--app', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 209, in <module>
    main(Operator)
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/main.py", line 394, in main
    charm = charm_class(framework)
  File "./src/charm.py", line 44, in __init__
    self.interfaces = get_interfaces(self)
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 263, in get_interfaces
    requires = {
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 264, in <dictcomp>
    name: SerializedDataInterface(
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 110, in __init__
    others = {
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 111, in <dictcomp>
    app.name: bag.get("_supported_versions")
  File "/usr/lib/python3.8/_collections_abc.py", line 660, in get
    return self[key]
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 400, in __getitem__
    return self._data[key]
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 384, in _data
    data = self._lazy_data = self._load()
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 748, in _load
    return self._backend.relation_get(self.relation.id, self._entity.name, self._is_app)
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 1351, in relation_get
    return self._run(*args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 1286, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: b'ERROR "" is not a valid unit or application\n'
2022-03-03 06:42:43 ERROR juju.worker.caasoperator.uniter.dex-auth/4.operation runhook.go:146 hook "oidc-client-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1
2022-03-03 06:42:43 INFO juju.worker.caasoperator.uniter.dex-auth/4 resolver.go:150 awaiting error resolution for "relation-broken" hook

dex-auth2.log
juju-crashdump-677b46e1-2d75-4ed2-887a-989e22e1d807.zip
kf-bundle.zip

[shayancanonical]

I ran into the same exception, below are a few observations on my deployment:

1. The following is a portion of the output of juju status:

App                        Version                    Status   Scale  Charm                    Store     Channel      Rev  OS          Address         Message
admission-webhook          res:oci-image@fc124ea      active       1  admission-webhook        charmhub  stable        12  kubernetes  10.152.183.145  
argo-controller            res:oci-image@0eec3c1      active       1  argo-controller          charmhub  stable        55  kubernetes                  
dex-auth                   res:oci-image@a74f783      active     3/2  dex-auth                 charmhub  2.28/stable   78  kubernetes  10.152.183.192  

...

Unit                          Workload     Agent  Address      Ports                 Message

...

dex-auth/6*                   error        idle   10.1.94.223  5556/TCP         hook failed: "oidc-client-relation-broken"
dex-auth/7                    waiting      idle   10.1.94.219  5556/TCP        Waiting for leadership
dex-auth/8                    waiting      idle   10.1.94.214  5556/TCP        Waiting for leadership

2. There are only two pods for dex-auth, both showing logs similar to:

time="2022-03-10T22:48:41Z" level=info msg="config using log level: debug"
time="2022-03-10T22:48:41Z" level=info msg="config issuer: http://10.64.140.43.nip.io/dex"
time="2022-03-10T22:48:41Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2022-03-10T22:48:41Z" level=info msg="creating custom Kubernetes resources"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource refreshtokens.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource passwords.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource passwords.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource offlinesessionses.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource connectors.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource connectors.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource devicerequests.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource devicerequests.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource devicetokens.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource devicetokens.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="config storage: kubernetes"
time="2022-03-10T22:48:41Z" level=info msg="config static client: Ambassador Auth OIDC"
time="2022-03-10T22:48:41Z" level=info msg="config connector: local passwords enabled"
time="2022-03-10T22:48:41Z" level=info msg="config skipping approval screen"
time="2022-03-10T22:48:41Z" level=info msg="listening (http) on 0.0.0.0:5556"

3. I am able to log into the kubeflow dashboard with the user/pass I set with juju config

[DomFleischmann]

Closing this issue in favor of #53 which contains more information