selkies-gstream hit 100% CPU

[mathico2]

Wishing you a Happy New Year! I've set up a node using Standard_NC16as_T4_v3 in AKS . However, we're encountering issue where the pod runs for a few minutes with CPU usage under 9% for selkies-gstream process , but then it abruptly jumps to over 100%, causing the pod to freeze again and become inaccessible . Please could you advise if there is anything that have been missing while deploying this apps within azure environment

[ehfd]

I'm not sure what might be an issue (I simply feel this might be something with the GStreamer backend), but try the new container that will be built in the next 10 minutes because there was a new Selkies-GStreamer release with a new GStreamer version patch.

[mathico2]

Hello ehfd-please could you provide URL for new release so I can update that image within deployment
I guess you refer to this below link

https://github.com/selkies-project/selkies-gstreamer/pkgs/container/selkies-gstreamer%2Fgstreamer

[mathico2]

@ehfd I used newly created selkies gstream docker image but unfortunately it doesn't even open the apps. Please could you advised which docker image need to push within Azure container registry then use that image for deployment

[ehfd]

No, you can use ghcr.io/selkies-project/nvidia-egl-desktop:22.04 or 20.04.
I built a new image with the new release.

[mathico2]

Ok thank you and I will let you know the outcome

…

On Fri, Jan 5, 2024, 2:04 PM Seungmin Kim ***@***.***> wrote: No, you can use ghcr.io/selkies-project/nvidia-egl-desktop:22.04 or 20.04. I built a new image with the new release. — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALJ6AOPXE5PY2JBSM3ZFZFLYNBFENAVCNFSM6AAAAABBNDSFQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZZGEZTIOBWGA> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[mathico2]

I just tried to use new docker image but still encountering same issue and this is deployment yaml file in use for this deployment within AKS :
apiVersion: apps/v1
kind: Deployment
metadata:
name: egl
spec:
replicas: 1
selector:
matchLabels:
app: egl
template:
metadata:
labels:
app: egl
spec:
hostname: egl
# Uncomment the below line to disable network isolation for WebRTC connectivity, may show an error if disallowed by the cluster
# hostNetwork: true
containers:
- name: egl
image: cazaw1232conregistry.azurecr.us/nvidia-egl-desktop:22.04
env:
- name: TZ
value: "UTC"
- name: SIZEW
value: "1920"
- name: SIZEH
value: "1080"
- name: REFRESH
value: "60"
- name: DPI
value: "96"
- name: CDEPTH
value: "24"
# Keep to default unless you know what you are doing with VirtualGL, VGL_DISPLAY should be set to either egl[n], or /dev/dri/card[n] only when the device was passed to the container
#- name: VGL_DISPLAY
# value: "egl"
# Choose either value: or secretKeyRef: but not both at the same time
- name: PASSWD
value: "mypasswd"
# valueFrom:
# secretKeyRef:
# name: my-pass
# key: my-pass
# Uncomment this to enable noVNC, disabing selkies-gstreamer and ignoring all its parameters except BASIC_AUTH_PASSWORD, which will be used for authentication with noVNC, BASIC_AUTH_PASSWORD defaults to PASSWD if not provided
# - name: NOVNC_ENABLE
# value: "true"
# Additional view-only password only applicable to the noVNC interface, choose either value: or secretKeyRef: but not both at the same time
# - name: NOVNC_VIEWPASS
# value: "mypasswd"
# valueFrom:
# secretKeyRef:
# name: my-pass
# key: my-pass
###
# selkies-gstreamer parameters, for additional configurations see lines that start with "parser.add_argument" in https://github.com/selkies-project/selkies-gstreamer/blob/master/src/selkies_gstreamer/__main__.py
###
# Change WEBRTC_ENCODER to x264enc, vp8enc, or vp9enc if you are using software fallback without allocated GPUs or your GPU doesn't support H.264 (AVCHD) under the NVENC - Encoding section in https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new
- name: WEBRTC_ENCODER
value: "nvh264enc"
- name: WEBRTC_ENABLE_RESIZE
value: "false"
- name: ENABLE_BASIC_AUTH
value: "true"
- name: ENABLE_HTTPS_WEB
value: "false"
# Volume mount trusted HTTPS certificate to new path for no web browser warnings
# - name: HTTPS_WEB_CERT
# value: /etc/ssl/certs/ssl-cert-snakeoil.pem
# - name: HTTPS_WEB_KEY
# value: /etc/ssl/private/ssl-cert-snakeoil.key
# Defaults to PASSWD if unspecified, choose either value: or secretKeyRef: but not both at the same time
# - name: BASIC_AUTH_PASSWORD
# value: "mypasswd"
# valueFrom:
# secretKeyRef:
# name: my-pass
# key: my-pass
###
# Uncomment below to use a TURN server for improved network compatibility
###
# - name: TURN_HOST
# value: "turn.example.com"
# - name: TURN_PORT
# value: "3478"
# Provide only TURN_SHARED_SECRET for time-limited shared secret authentication or both TURN_USERNAME and TURN_PASSWORD for legacy long-term authentication, but do not provide both authentication methods at the same time
# - name: TURN_SHARED_SECRET
# valueFrom:
# secretKeyRef:
# name: turn-shared-secret
# key: turn-shared-secret
# - name: TURN_USERNAME
# value: "username"
# Choose either value: or secretKeyRef: but not both at the same time
# - name: TURN_PASSWORD
# value: "mypasswd"
# valueFrom:
# secretKeyRef:
# name: turn-password
# key: turn-password
# Change to tcp if the UDP protocol is throttled or blocked in your client network, or when the TURN server does not support UDP
# - name: TURN_PROTOCOL
# value: "udp"
# You need a valid hostname and a certificate from authorities such as ZeroSSL (Let's Encrypt may have issues) to enable this
# - name: TURN_TLS
# value: "false"
stdin: true
tty: true
ports:
- name: http
containerPort: 8080
protocol: TCP
resources:
limits:
memory: 64Gi
cpu: "16"
nvidia.com/gpu: 1
requests:
memory: 100Mi
cpu: 100m
volumeMounts:
- mountPath: /dev/shm
name: dshm
- mountPath: /cache
name: egl-cache-vol
- mountPath: /home/user
name: egl-root-vol
- mountPath: /dev/dri
name: drm
tolerations:
- key: "sku"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
volumes:
- name: dshm
emptyDir:
medium: Memory
- name: egl-cache-vol
emptyDir: {}
# persistentVolumeClaim:
# claimName: egl-cache-vol
- name: egl-root-vol
emptyDir: {}
# persistentVolumeClaim:
# claimName: egl-root-vol
- name: drm
emptyDir: {}

[ehfd]

Can you follow the procedures in: https://github.com/selkies-project/selkies-gstreamer#install-the-packaged-version-on-a-standalone-machine-or-cloud-instance outside Kubernetes or any containers in the same VM instance? I want to check if it's a hardware issue or a container issue.

[ehfd]

Similar condition (both on Azure): selkies-project/docker-nvidia-glx-desktop#50

[ehfd]

@justinbowes

Selkies-GStreamer directly goes to NVENC so I doubt this is VirtualGL. Do you have any leads? It's just because you were also using Azure's GPUs.

[justinbowes]

@mathico2 On Azure, for virtual workstation applications you might try GRID driver (which is supported on the T4 instances -- see the exception here https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup ).

Beyond that, I'd be trying to isolate the layer in which the issue occurs. What is the output of nvidia-smi encodersessions and nvidia-smi dmon, outside of containers, while this is happening?

Also worth checking the kernel messages to see if the driver is complaining.

[mathico2]

Thanks, I'll check it out.

…

On Fri, Feb 2, 2024, 7:57 PM Justin Bowes ***@***.***> wrote: @mathico2 <https://github.com/mathico2> On Azure, for virtual workstation applications you might try GRID driver (which is supported on the T4 instances -- see the exception here https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup ). Beyond that, I'd be trying to isolate the layer in which the issue occurs. What is the output of nvidia-smi encodersessions and nvidia-smi dmon while this is happening? Also worth checking the kernel messages to see if the driver is complaining. — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALJ6AOOUBMGCUY2NRB7RH3DYRUZH3AVCNFSM6AAAAABBNDSFQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRUGQ4TONBXGE> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[ehfd]

I agree that there should be comparison inside and outside the container to progress further.

[ehfd]

@mathico2 Any follow ups?

[mathico2]

Hello Still not able to have that work. /r

…

On Tue, Mar 12, 2024, 8:18 AM Seungmin Kim ***@***.***> wrote: @mathico2 <https://github.com/mathico2> Any follow ups? — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALJ6AOKAV6QJOOGQ5WTLYXLYX3XADAVCNFSM6AAAAABBNDSFQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGUZDANZQGE> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[ehfd]

https://github.com/selkies-project/selkies-gstreamer#install-the-packaged-version-on-a-standalone-machine-or-cloud-instance

I need you to follow the procedures here outside of a container within the same instance to continue.

[mathico2]

Ok, I will do it.

…

On Sun, Mar 24, 2024 at 11:48 AM Seungmin Kim ***@***.***> wrote: https://github.com/selkies-project/selkies-gstreamer#install-the-packaged-version-on-a-standalone-machine-or-cloud-instance I need you to follow the procedures here outside of a container within the same instance to continue. — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALJ6AOJPLC5OGAQN53IV2QTYZ3YUXAVCNFSM6AAAAABBNDSFQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJWHA2TAMBZGY> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[ehfd]

Leads to: python-xlib/python-xlib#242

[ehfd]

Just to confirm: did you happen to use an international keyboard layout? @mathico2

[mathico2]

No

…

On Sun, May 5, 2024, 9:15 AM Seungmin Kim ***@***.***> wrote: Just to confirm: did you happen to use an international keyboard layout? @mathico2 <https://github.com/mathico2> — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALJ6AOJVFSGL4TT6HJ2IWX3ZAYWIRAVCNFSM6AAAAABBNDSFQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJUHAYDMNBYGA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>