Possible configuration issue, unable to get logs

[paul42]

Environment details:

used k3s (from rancher) to set up 2 node kubernetes cluster on centos7 vms, remote.

Steps to reproduce the issue:

1. use k3s create a cluster with two nodes
2. use helm to install chart for openwhisk
3. label one node with 'core' and one with invoker

Provide the expected results and outputs:

expected: wsk activation logs -l would have most recent logs from invokes
expected:

Provide the actual results and outputs:

in the hello.js example I'm unable to get any output from a console.log('test') as well as dotnet core examples

Additional information you deem important:

this is a deployment that is not on your standard cloud provider like aws/azure/gke, so is there anything I can check to see where logs are saved?

I noticed on another issue about logs that the command wskadmin syslog get was mentioned, I tried the following and got no such files:

kubectl -n openwhisk -ti exec owdev-wskadmin -- wskadmin syslog get -g *
grep: /var/log/dotnet-install.sh/dotnet-install.sh_logs.log: No such file or directory
grep: /var/log/getRuntimeAction.js/getRuntimeAction.js_logs.log: No such file or directory
grep: /var/log/hello.js/hello.js_logs.log: No such file or directory
grep: /var/log/helloDotNet.zip/helloDotNet.zip_logs.log: No such file or directory
grep: /var/log/helloWorldAction.js/helloWorldAction.js_logs.log: No such file or directory

sorry if this is a bit of a stretch, but are there any values I need to change on my helm charts to fix where it puts things? or should I just do a helm uninstall and reinstall and try again? the invoking and uploading of code is working great so far, just... no logs?

[dgrove-oss]

It's possible that we have the same issue with k3s as we do with OpenShift 4.5 (apache/openwhisk-deploy-kube#665): the underlying container runtime doesn't support the json logs interface we are trying to use. This is just a semi-educated guess; I have not looked at k3s in enough detail to know how it does logging.

[paul42]

hey @dgrove-oss thanks for the info - I'll poke around a bit - do you know if using kubeadm to make a cluster does work for logs? Also is logging a value in the helm chart I can poke at, or is it more complex? Either way thanks for taking a look, I appreciate it

[paul42]

just as an aside, my most recent effort was:

• trying microk8s.io instead of k3s
• installed openEBS which came with mk8s it so I'd have a storageclass
• for values.yaml, input

invoker:
  options: "-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.LogDriverLogStoreProvider"

next step will be trying to do a manual cluster build with kubeadm, et. al and see if that fixes anything

[paul42]

I'm also noticing on some tutorials for setting up kubernetes clusters with kubeadm, they suggest in a /etc/docker/daemon.json to have the following:

sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

do any of these options (log-driver as json-file) or cgroup driver of systemd result in activation logs not showing up?

[paul42]

as an aside building a manual cluster with kubeadm didn't fix either:

1. Using weave-net as cni
2. using containerd as container runtime.

going to try for kubeadm / weave / docker with docker config above next

[paul42]

after building my own cluster with KubeAdm:

1. Weave Net (Ip Allocation of 192.168.0.0/24)
2. Docker Container Runtime (with Json-file logging in daemon.json)
3. very basic openEbs install (helm chart, and using openebs-hostpath as storage class

I'm now able to get logs from activations and I'm feeling good, thanks for the help on this