live-0-6dcff4f549-rvm6h pod failed occasionally

[haodengeke]

HW: ICE,SKY
OS: CentOS 8.3.2011,RHEL 8.3, ubuntu,
Case:CIR_BMRA_RegionDC_CDN_Xeon
Command：ansible-playbook -i cdn_inventory.ini playbooks/cir.yml --extra-vars "profile=basic"

Result:
kubectl get pod -A*
NAMESPACE NAME READY STATUS RESTARTS AGE
default cdn-8668b855ff-7rqrt 1/1 Running 0 106s
default kafka-5b4b8845bc-wsv5b 1/1 Running 1 106s
default live-0-6dcff4f549-rvm6h 0/1 CrashLoopBackOff 4 105s
default redis-54fbcb48dd-ttb98 1/1 Running 0 106s
default vod-0-b7cddccdd-qlxrn 1/1 Running 0 106s
default vod-0-b7cddccdd-zfrhc 1/1 Running 0 106s
default zookeeper-674686784d-dz4xt 1/1 Running 0 106s
kube-system coredns-7677f9bb54-h8wck 1/1 Running 2 29m
kube-system coredns-7677f9bb54-qkjrw 1/1 Running 1 29m
kube-system dns-autoscaler-5b7b5c9b6f-hv6hd 1/1 Running 1 29m
kube-system docker-registry-docker-registry-545dd99446-5x8p4 2/2 Running 0 26m
kube-system kube-apiserver-aq09-00-wc 1/1 Running 0 106s
kube-system kube-controller-manager-aq09-00-wc 1/1 Running 4 31m
kube-system kube-flannel-jm7hf 1/1 Running 1 30m
kube-system kube-flannel-qhwmr 1/1 Running 1 30m
kube-system kube-multus-ds-amd64-52pm6 1/1 Running 1 29m
kube-system kube-multus-ds-amd64-hbmqf 1/1 Running 1 29m
kube-system kube-proxy-qmfjp 1/1 Running 1 30m
kube-system kube-proxy-sgqwt 1/1 Running 2 30m
kube-system kube-scheduler-aq09-00-wc 1/1 Running 1 19m
kube-system kubernetes-dashboard-86c6f9df5b-pvgxn 1/1 Running 2 29m
kube-system kubernetes-metrics-scraper-678c97765c-9fm99 1/1 Running 2 29m
kube-system nginx-proxy-ar09-07-cyp 1/1 Running 2 30m
kube-system node-feature-discovery-controller-5bf5c49849-q4knn 1/1 Running 0 22m
kube-system node-feature-discovery-worker-skh7d 1/1 Running 0 22m
monitoring collectd-tn28r 3/3 Running 0 18m
monitoring node-exporter-c4bw8 2/2 Running 0 21m
monitoring node-exporter-fdgmx 2/2 Running 0 21m
monitoring prometheus-k8s-0 5/5 Running 1 21m
monitoring prometheus-operator-5bb8cb9d8f-4nkd9 2/2 Running 0 21m
monitoring tas-telemetry-aware-scheduling-575ccbc9d4-p5dd4 2/2 Running 4 19m

Error log for this issue:
Events:
Type Reason Age From Message

---

Warning FailedScheduling 47s default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
Warning FailedScheduling 47s default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 45s default-scheduler Successfully assigned default/live-0-6dcff4f549-rj7g2 to av09-07-wp
Normal AddedInterface 44s multus Add eth0 [10.244.1.25/24]
Normal Pulled 24s (x3 over 44s) kubelet Container image "tc_transcode_xeon:latest" already present on machine
Normal Created 24s (x3 over 44s) kubelet Created container live-0
Normal Started 24s (x3 over 43s) kubelet Started container live-0
Warning BackOff 10s (x4 over 39s) kubelet Back-off restarting failed container

[root@av09-06-wp build]# kubectl logs live-0-6dcff4f549-rj7g2
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr/local --libdir=/usr/local/lib --enable-shared --disable-static --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --extra-cflags=-w --enable-gpl --enable-nonfree --enable-libass --enable-libfreetype --disable-xlib --disable-sdl2 --disable-hwaccels --disable-vaapi --enable-libv4l2 --enable-indev=v4l2 --enable-libopus --enable-libvpx --enable-libvorbis --enable-libx264 --enable-libx265 --enable-libsvtav1 --enable-libsvthevc --enable-libsvtvp9 --enable-libdav1d
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/var/www/archive/3115738.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isomavc1
creation_time : 2019-10-22T12:48:08.000000Z
Duration: 00:00:18.79, start: 0.000000, bitrate: 25358 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160, 25102 kb/s, 24 fps, 24 tbr, 24 tbn, 48 tbc (default)
Metadata:
creation_time : 2019-10-22T12:48:08.000000Z
handler_name : L-SMASH Video Handler
encoder : AVC Coding
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 253 kb/s (default)
Metadata:
creation_time : 2019-10-22T12:48:08.000000Z
handler_name : L-SMASH Audio Handler
rtmp://cdn-service/dash/media_0_1: Input/output error

[lvcarry]

make parameters：
cmake -DSCENARIO=cdn -DPLATFORM=Xeon -DNVODS=2 -DNLIVES=1 -DREGISTRY= /usr/src/cdn-transcode

[lvcarry]

we meet this issue in high reproduce rate.
Pls help debug this issue.

[mnoiy]

The cause of this bug has been found.
When deploying a CDN project on multiple machines environment, command: ‘make volume’ will transfer the video to the corresponding folder on working nodes, the user on the working node must have the same user as the user who deploys the CDN on the master node.
If CDN is deployed as the root user on the master node, the working node must also use root privileges to receive video files. Therefore, the permission of the video storage folder :home/username/sdp/cache/video/ pushed to the working node is belong to root user. However, docker does not have root permission, so it is impossible to read the video file from the folder, which causes an input/output error.
Want to avoid permission issues:
① Use a non-root user with the same name on each machine
or
②Use ‘chown -R -v [non-root user] [name of the video folder pushed on the working node]’ command to change the owner of the cache folder on the worker node to a non-root user.

But after solving this problem, We found that there is a certain probability that a new problem will occur in the deployment of CDN in a multi-machine environment ：
Error writing trailer of rtmp://cdn-service/dash/media_0_1: Broken pipe

Result:
root@skl-yu02:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
cdn-8668b855ff-9chpm 1/1 Running 0 98m
kafka-5b4b8845bc-bz4wf 1/1 Running 0 98m
live-0-5c4885b76c-kr42h 0/1 CrashLoopBackOff 13 ( ago) 98m
redis-54fbcb48dd-ssh8c 1/1 Running 0 98m
xcode-0-585c7fbb-7mrvm 1/1 Running 0 98m
xcode-0-585c7fbb-g8djn 1/1 Running 0 98m
zookeeper-674686784d-kpnn4 1/1 Running 0 98m

error pod logs：
root@skl-yu02:/home/bihuanxu/wxs/CDN-Transcode-Sample/build# kubectl logs live-0-5c4885b76c-4x7bg
ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr/local --libdir=/usr/local/lib --enable-shared --disable-static --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --extra-cflags=-w --enable-gpl --enable-nonfree --enable-libass --enable-libfreetype --disable-xlib --disable-sdl2 --disable-hwaccels --disable-vaapi --enable-libv4l2 --enable-indev=v4l2 --enable-libopus --enable-libvpx --enable-libvorbis --enable-libx264 --enable-libx265 --enable-libsvtav1 --enable-libsvthevc --enable-libsvtvp9 --enable-libdav1d
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/var/www/archive/3115738.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isomavc1
creation_time : 2019-10-22T12:48:08.000000Z
Duration: 00:00:18.79, start: 0.000000, bitrate: 25358 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 3840x2160, 25102 kb/s, 24 fps, 24 tbr, 24 tbn, 48 tbc (default)
Metadata:
creation_time : 2019-10-22T12:48:08.000000Z
handler_name : L-SMASH Video Handler
encoder : AVC Coding
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 253 kb/s (default)
Metadata:
creation_time : 2019-10-22T12:48:08.000000Z
handler_name : L-SMASH Audio Handler
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:0 -> #1:0 (h264 (native) -> hevc (libsvt_hevc))
Press [q] to stop, [?] for help
[libx264 @ 0x55869808b8c0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x55869808b8c0] profile High, level 3.1, 4:2:0, 8-bit
[libx264 @ 0x55869808b8c0] 264 - core 161 - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=1 ref=2 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=2 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=12 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=2 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=100 keyint_min=10 scenecut=40 intra_refresh=0 rc_lookahead=10 rc=abr mbtree=1 bitrate=8000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, flv, to 'rtmp://cdn-service/hls/media_0_0':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isomavc1
encoder : Lavf58.29.100
Stream #0:0(und): Video: h264 (libx264) ([7][0][0][0] / 0x0007), yuv420p, 856x480, q=-1--1, 8000 kb/s, 30 fps, 1k tbn, 30 tbc (default)
Metadata:
creation_time : 2019-10-22T12:48:08.000000Z
handler_name : L-SMASH Video Handler
encoder : Lavc58.54.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/8000000 buffer size: 0 vbv_delay: -1
SVT [version]: SVT-HEVC Encoder Lib v1.5.0
SVT [build] : GCC 7.5.0 64 bit
LIB Build date: Mar 23 2021 11:03:42
-------------------------------------------
Number of logical cores available: 8
Number of PPCS 165
-------------------------------------------
SVT [config]: Main Profile Tier (auto) Level (auto)
SVT [config]: EncoderMode / Tune : 9 / 1
SVT [config]: EncoderBitDepth / CompressedTenBitFormat / EncoderColorFormat : 8 / 0 / 1
SVT [config]: SourceWidth / SourceHeight / InterlacedVideo : 856 / 480 / 0
SVT [config]: Fps_Numerator / Fps_Denominator / Gop Size / IntraRefreshType : 30 / 1 / 100 / 1
SVT [config]: HierarchicalLevels / BaseLayerSwitchMode / PredStructure : 3 / 0 / 2
SVT [config]: BRC Mode / QP / LookaheadDistance / SceneChange : CQP / 32 / 17 / 1
SVT [config]: BitRateReduction / ImproveSharpness : 0 / 0
SVT [config]: tileColumnCount / tileRowCount / tileSliceMode / Constraint MV : 1 / 1 / 0 / 0
SVT [config]: De-blocking Filter / SAO Filter : 1 / 1
SVT [config]: HME / UseDefaultHME : 1 / 1
SVT [config]: MV Search Area Width / Height : 16 / 7
SVT [config]: HRD / VBV MaxRate / BufSize / BufInit : 0 / 0 / 0 / 90
-------------------------------------------
Output #1, flv, to 'rtmp://cdn-service/dash/media_0_1':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: mp42mp41isomavc1
encoder : Lavf58.29.100
Stream #1:0(und): Video: hevc (libsvt_hevc) ([12][0][0][0] / 0x000C), yuv420p, 856x480, q=10-48, 8000 kb/s, 30 fps, 1k tbn, 30 tbc (default)
Metadata:
creation_time : 2019-10-22T12:48:08.000000Z
handler_name : L-SMASH Video Handler
encoder : Lavc58.54.100 libsvt_hevc
av_interleaved_write_frame(): Broken pipe 665kB time=00:00:00.43 bitrate=12555.5kbits/s speed=0.0764x
[flv @ 0x55869808a640] Failed to update header with correct duration.
[flv @ 0x55869808a640] Failed to update header with correct filesize.
[flv @ 0x5586980daa00] Failed to update header with correct duration.
[flv @ 0x5586980daa00] Failed to update header with correct filesize.
Error writing trailer of rtmp://cdn-service/dash/media_0_1: Broken pipe
frame= 40 fps=5.3 q=-1.0 Lq=-0.0 size= 1578kB time=00:00:01.53 bitrate=8426.2kbits/s speed=0.205x
video:1586kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[libx264 @ 0x55869808b8c0] frame I:1 Avg QP: 6.85 size:140374
[libx264 @ 0x55869808b8c0] frame P:14 Avg QP: 9.51 size: 65756
[libx264 @ 0x55869808b8c0] frame B:25 Avg QP:12.39 size: 22020
[libx264 @ 0x55869808b8c0] consecutive B-frames: 5.0% 5.0% 90.0%
[libx264 @ 0x55869808b8c0] mb I I16..4: 7.0% 15.4% 77.6%
[libx264 @ 0x55869808b8c0] mb P I16..4: 0.9% 1.6% 0.9% P16..4: 62.2% 12.1% 10.9% 0.0% 0.0% skip:11.3%
[libx264 @ 0x55869808b8c0] mb B I16..4: 0.1% 0.0% 0.0% B16..8: 15.2% 5.1% 2.8% direct:39.9% skip:36.9% L0:25.8% L1:41.5% BI:32.7%
[libx264 @ 0x55869808b8c0] final ratefactor: 9.39
[libx264 @ 0x55869808b8c0] 8x8 transform intra:26.0% inter:34.9%
[libx264 @ 0x55869808b8c0] coded y,uvDC,uvAC intra: 92.7% 99.3% 99.0% inter: 52.1% 31.6% 22.0%
[libx264 @ 0x55869808b8c0] i16 v,h,dc,p: 40% 40% 7% 13%
[libx264 @ 0x55869808b8c0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 4% 60% 13% 3% 2% 1% 7% 2% 8%
[libx264 @ 0x55869808b8c0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 38% 13% 4% 6% 2% 17% 2% 10%
[libx264 @ 0x55869808b8c0] i8c dc,h,v,p: 41% 41% 5% 13%
[libx264 @ 0x55869808b8c0] Weighted P-Frames: Y:14.3% UV:0.0%
[libx264 @ 0x55869808b8c0] ref P L0: 85.4% 14.6%
[libx264 @ 0x55869808b8c0] ref B L0: 77.0% 23.0%
[libx264 @ 0x55869808b8c0] kb/s:7735.04
Conversion failed!