Periscope does not publish images for Windows Server 2022 nodes

[sumicalbin]

Describe the bug
From the discussions I've been having it seems that this script, collectlogs, or even other logging scripts can be included in periscope. It is unclear whether this has been added as a customizable configuration on the yaml or is preset somewhere.
I am curious if this has been included and if there is an example for this?
To Reproduce
When I run periscope with the latest build, I do not see a collectlogs output.
The pod logs

Part of my kustomization.yaml

Desktop (please complete the following information):

• OS: windows
• Version: 2022

[sumicalbin]

I also tried with az aks kollect
my command az aks kollect -g $myResourceGroup -n $myAKSCluster --storage-account $storageAccountName --sas-token $sasKey
And the output that was printed

[peterbom]

Hi @sumicalbin - thank you for reporting this. I think you've highlighted a few issues here.

First, running the collectlogs.ps1 script is currently not supported. Windows diagnostics uses the collect-windows-logs.ps1 script. Is that not sufficient for your needs?

Periscope does allow collection of arbitrary log files based on DIAGNOSTIC_NODELOGS_LIST_WINDOWS environment variable, but it doesn't allow users to configure scripts to run to generate those files. A workaround might be to run the script on the nodes yourself, then use the above env var to collect the resulting log files in Periscope. Would that be feasible?

Some other notes:

• I think the ImagePullBackOff for the pod running on the system node pool is because the kustomization.yaml doesn't specify the Linux image (for name: periscope-linux).
• I'm unsure why the Periscope pod on the Windows node is crashing (both when run manually and when using the kollect command). Is there any hint in the pod logs?
• When you deploy Periscope using kustomize, you need to explicitly enable the Windows HPC feature, like this (there may be a documentation gap here), which is why you're not seeing the diagnostics-collection-win pod that runs collect-windows-logs.ps1:

components:
- https://github.com/Azure/aks-periscope//deployment/components/win-hpc?ref=0.0.11

[peterbom]

Hi @sumicalbin - I'm reliably informed that collect-windows-logs.ps1 calls collectlogs.ps1, so if we can get that feature running correctly you should be able to get the logs you need.

The main blocker looks like the Periscope pod (on the Windows node) crashing, so I'd start by looking at the logs for that (kubectl logs -n aks-periscope aks-periscope-win-xxxxx [--previous]).

[sumicalbin]

@peterbom Thanks for the update regarding "enable the Windows HPC feature" that allowed me to use the yaml.

I noticed two different issues.

1. The blob storage sasToken seems to be causing some issue. I have generated the token a few different times, but I'm not able to find the exact reason as to why the token isn't working. I'll keep trying this one.
2. When I run kubectl logs -n aks-periscope aks-periscope-win-xxxxx --pervious I do not see any logs.
   Running kubectl describe pods -n aks-periscope aks-periscope-win-xxxxx provides me with the event log below. It seems to indicate a mismatch between the container and the host os.
   
   I tried docker inspect mcr.microsoft.com/aks/periscope:0.0.11 but couldn't see the windows version.
   My host version is
   akswin22000000 Ready agent 42d v1.23.12 10.224.0.103 <none> Windows Server 2022 Datacenter 10.0.20348.1249 containerd://1.6.8+azure

[peterbom]

Great observation, @sumicalbin and thank you for alerting us to this.

We don't currently support deploying to Windows Server 2022 nodes (the images are built for 2019). I've edited the title accordingly.

Unfortunately, I'm not sure of a workaround for you at the moment. We'll discuss what's involved in enabling Periscope for Server 2022 and update here soon. cc: @Tatsinnit

[peterbom]

Hi @sumicalbin: could you try using the following?

images:
- name: periscope-linux
  newName: mcr.microsoft.com/aks/periscope
  newTag: 0.0.12
- name: periscope-windows
  newName: mcr.microsoft.com/aks/periscope
  newTag: 0.0.12

Note that both images now reference the same image tag, which refers to a multi-platform manifest. The periscope-win image is not needed.

When we initially deployed v0.0.12, there was an error uploading the manifest, resulting in the Windows 2022 image getting excluded. We've now re-run the release, which should have resolved any image-pull errors you may have observed if you'd already tried this on Windows 2022 nodes in the past few days.

[sumicalbin]

@peterbom Still seem to have an issue pulling from mcr. When I got to the site and check the tags I see 0.0.12 missing

[peterbom]

Hi @sumicalbin - that looks like the tag list for the periscope-win image. Try this instead (see note above about cross-platform images using the same tag).

[sumicalbin]

HI @peterbom, yeah that was the problem. Was still using the periscope-win tag list.

Pulling the image works for windows2022 now! Thanks for updating that.

I did notice that the windows2019 will have the Collect-Windows-Logs directory, however, the windows2022 is missing that directory. I also tried running only windows2022 and the Collect-Windows-Logs directory is still missing. The yaml settings were not different for either run.

[peterbom]

Hi @sumicalbin - I'm noticing a similar issue on my own cluster, but I'm still struggling to reproduce it reliably. Could you run kubectl logs -n aks-periscope diagnostics-collection-win2022-xxxxx for whichever pod is running on a 2022 node? In my case the last log entry says Data collection successful; output = filters.xml (there should be a bunch more stuff, but for me it's hanging when a nested powershell script runs hcsdiag list, and there's no more output after that). Do you have the same symptom?

Also, can you share the Kubernetes version and VM SKU of your Windows 2022 node pool?

[sumicalbin]

Hey @peterbom Here is the output for the logs on diagnostics-collection-win2022-xxxxx

I don't see the Data collection successful; output anywhere

Here is the version info
clientVersion: buildDate: "2022-08-23T17:44:59Z" compiler: gc gitCommit: a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2 gitTreeState: clean gitVersion: v1.25.0 goVersion: go1.19 major: "1" minor: "25" platform: windows/amd64 kustomizeVersion: v4.5.7 serverVersion: buildDate: "2022-12-13T08:17:45Z" compiler: gc gitCommit: 9710807c82740b9799453677c977758becf0acbb gitTreeState: clean gitVersion: v1.24.9 goVersion: go1.18.9 major: "1" minor: "24" platform: linux/amd64

Windows2022: standard D4s V3