Charm build fails ('NoneType' object is not subscriptable)

[rs22]

My goal is to install kubeflow 1.0 on microk8s, which seems to use this bundle when running microk8s.enable kubeflow. From what I can tell, the master branch already supports kubeflow 1.0, but the current edge version of microk8s in the snap store seems to install an older version of the bundle, even when using KUBEFLOW_CHANNEL=edge. Moreover, when I run the deployment command using --channel=edge from within this repository (as per the README), I'm still getting a pre-1.0 version of kubeflow.

My environment consists of Ubuntu 18.04.4 LTS (amd64), microk8s 1.17/stable; I've followed the README (though juju-helpers required --edge to install) and I additionally installed 'pyyaml' via pip3 and 'charm' via snap.

Instead of downloading charms from the store, I now tried to use the --build option for deployment. The entire deployment command is: python3 scripts/cli.py deploy-to uk8s --build --public-address <machine_name>. It fails with this error message:

Traceback (most recent call last):
  File "/snap/charm/440/bin/charm-build", line 11, in <module>
    load_entry_point('charm-tools==2.7.3', 'console_scripts', 'charm-build')()
  File "/snap/charm/440/lib/python3.6/site-packages/charmtools/build/builder.py", line 941, in main
    build()
  File "/snap/charm/440/lib/python3.6/site-packages/charmtools/build/builder.py", line 649, in __call__
    self.generate()
  File "/snap/charm/440/lib/python3.6/site-packages/charmtools/build/builder.py", line 592, in generate
    layers = self.fetch()
  File "/snap/charm/440/lib/python3.6/site-packages/charmtools/build/builder.py", line 272, in fetch
    self.post_metrics('charm', self.name, False)
  File "/snap/charm/440/lib/python3.6/site-packages/charmtools/build/builder.py", line 529, in post_metrics
    cid = conf['cid']
TypeError: 'NoneType' object is not subscriptable

[knkski]

One thing to note is that support for upstream's 1.0 jupyter notebooks is blocked on supporting Istio, which is work that is in-progress. Other than that, this bundle should match upstream kubeflow 1.0, and where it doesn't should be filed as a bug.

That aside, the error that you're getting looks like it stems from having a weird value in ~/.config/charm-build.conf. Can you post the contents of that file? And does the charm build if you just rename/remove the existing file?

[knkski]

Also, what version of charm are you using (snap info charm)? I had some issues with 2.7.3, and had to revert to 2.7.2.

[rs22]

Thanks for your help! Renaming my old ~/.config/charm-build.conf fixed it. Its contents were:

{cid: f1912290-64a8-4365-9c89-5a2a1f1fc7b1}

The newly created charm-build.conf looks very similar, though:

{cid: 35a57dd2-06e8-4e58-93fe-33b17439229e}

I'm using charm 2.7.3, which is the current version of the snap.

The deployment now proceeds until the juju wait -wv -m kubeflow step, which errors because of katib-manager (ERROR:root:katib-manager/0 failed: workload status is error).

# juju status katib-manager
Model     Controller  Cloud/Region        Version  SLA          Timestamp
kubeflow  uk8s        microk8s/localhost  2.7.4    unsupported  09:50:14+01:00

App            Version  Status  Scale  Charm          Store  Rev  OS          Address  Notes
katib-manager           error       1  katib-manager  local    0  kubernetes           hook failed: "mysql-relation-changed"

Unit              Workload  Agent  Address  Ports  Message
katib-manager/0*  error     idle                   hook failed: "mysql-relation-changed"

Is this a known issue? I don't want to bother you with this, when I should probably wait until the bundle is officially released for kubeflow 1.0.

[knkski]

@rs22: Can you try redeploying and seeing if that fixes the issue with katib-manager? And if it happens again, post the output from microk8s.kubectl logs -n kubeflow --tail 1000 -l juju-operator=katib-manager (probably in a new comment, since it doesn't look to be related)?