Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Jetson nano support #6

Closed
fnoop opened this issue Feb 9, 2020 · 8 comments
Closed

Add Jetson nano support #6

fnoop opened this issue Feb 9, 2020 · 8 comments

Comments

@fnoop
Copy link
Member

fnoop commented Feb 9, 2020

Jetson nano CSI port has a special source

@Davidsastresas
Copy link

Davidsastresas commented Feb 10, 2020

It seems the current prefered gstreamer source for jetson is nvarguscamerasrc, as stated in:

https://developer.download.nvidia.com/embedded/L4T/r32_Release_v1.0/Docs/Accelerated_GStreamer_User_Guide.pdf?LQrRPdv8wWDcwB3wjVjCdvk9oNCGOvjmwi2Yz7GrbyY5q76kQ96_YRsiJCd5eAAUSijWTRkLrKxxSa8BFeaEJ42kqZyBg22o-LZi608wN-xWMnkjlXaHj54ilgu-2VUjfkzeHPcDFVbdnWePDtmJ5xtg-bbUmhyDgqfpbKGLXs7nu7G9nto

An example of a working pipeline with imx219 as camera source and encoding h264 and sending it by UDP would be:

gst-launch-1.0 -e nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1' ! nvv4l2h264enc bitrate=8000000 insert-sps-pps=true ! rtph264pay mtu=1400 ! udpsink host=$CLIENT_IP port=5000 sync=false

which can be seen on the receiver ( tested on ubuntu 18 ) with:

gst-launch-1.0 udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! queue ! avdec_h264 ! glimagesink

There are some reference pipelines in this link:

https://developer.ridgerun.com/wiki/index.php?title=Jetson_Nano/Gstreamer/Example_Pipelines/Streaming

Also, there may be some interesting features to add from the VisionWorks library examples, particularly video stabilizaiton. It seems it can do 720p almost realtime, although some modifications to the code should be needed to remove jitter and latency, removing old frames in case the current one takes too long. It can vary a bit depending on the sceene. This repo has the VisionWorks examples modified for taking the imx219 input, by thefault they are meant to be used with a video file:

https://github.com/Davidsastresas/VisionWorks

Under samples folder there are instructions to build them. Once build, the examples should be in:

/VisionWorks-1.6-Samples/bin/aarch64/linux/release

and the video stabilization sample could be run like:

./nvx_demo_video_stabilizer --source="device:///nvcamera" -n 3 --crop=0.2

where n are the number of frames taken for the stabilization and crop the amount of frame it is cropped for the stabilization. The number of frames taken shall be adjusted to get a good balance between stabilization quality and latency. This command will take the imx219 as input and will show in screen 2 windows with the stabilized and non stabilized video, showing the crop as a rectangle in the non stabilized video.

@fnoop
Copy link
Member Author

fnoop commented Feb 10, 2020

visiond detects nanon/rpicam as:

2020-02-10 21:15:11,634 - Visiond - DEBUG - V4l2 device input: Camera 0:2
2020-02-10 21:15:11,635 - Visiond - INFO - v4l2 device /dev/video0 is a camera, autoselecting
2020-02-10 21:15:11,636 - Visiond - DEBUG - V4l2 device input: Camera 0:2
2020-02-10 21:15:11,636 - Visiond - DEBUG - driver: tegra-video
2020-02-10 21:15:11,637 - Visiond - DEBUG - card: vi-output, imx219 6-0010
2020-02-10 21:15:11,638 - Visiond - DEBUG - Camera format: 10-bit Bayer RGRG/GBGB
2020-02-10 21:15:11,639 - Visiond - DEBUG - Format: yuv : 10-bit Bayer RGRG/GBGB
2020-02-10 21:15:11,640 - Visiond - DEBUG - Format: mjpeg : 10-bit Bayer RGRG/GBGB
2020-02-10 21:15:11,640 - Visiond - DEBUG - Format: h264 : 10-bit Bayer RGRG/GBGB
2020-02-10 21:15:11,640 - Visiond - CRITICAL - Error constructing pipeline: ValueError('Error detecting camera video format'), retrying in 10 sec

So need to parse driver+card for tegra-video/imx219 and if so then switch the input to nvarguscamerasrc. Input is currently hardcoded to use config.args.input which is specifically set in the default config to input=v4l2src. Instead, autodetect if nvarguscamerasrc is required, otherwise use v4l2src as the default, and allow the config to override either.

@fnoop
Copy link
Member Author

fnoop commented Feb 11, 2020

Jetson floods syslog with:

Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:16:51 maverick-nano nvargus-daemon[5676]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)

@fnoop
Copy link
Member Author

fnoop commented Feb 11, 2020

New RTSP connection always fails with:

[dev] [mav@maverick-nano /var/log]$ sudo journalctl -u nvargus-daemon
-- Logs begin at Tue 2020-02-11 10:34:08 GMT, end at Tue 2020-02-11 10:52:04 GMT. --
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: === NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== python3[26122]: Connection established (7FA613B1D0)OF
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: ---- imager: No override file found. ----
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: LSC: LSC surface is not based on full res!
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: === python3[26122]: CameraProvider initialized (0x7fa0836070)LSC: LSC surface is not based on full res!
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMu
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: updateOutputSettings: Set Control failed. Use cached values
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMu
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: updateOutputSettings: Set Control failed. Use cached values
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl()
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMu
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: updateOutputSettings: Set Control failed. Use cached values
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvCamV4l2) Error IoctlFailed:  (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMu
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: (NvOdmDevice) Error IoctlFailed:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi
Feb 11 10:51:48 maverick-nano nvargus-daemon[23005]: updateOutputSettings: Set Control failed. Use cached values
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: SCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 478)
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: SCF: Error Timeout:  (propagating from src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 519)
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: SCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 561)
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: SCF: Error Timeout:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: SCF: Error Timeout: Worker thread IspHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: Error: waitCsiFrameStart timeout guid 1
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: ************VI/CSI Debug Registers**********
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: VI_CFG_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: VI_CFG_INTERRUPT_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: VI_CSI_0_ERROR_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: VI_CSI_0_ERROR_INT_MASK_0 = 0x0000001f
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: VI_CSI_1_ERROR_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: VI_CSI_1_ERROR_INT_MASK_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_A_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_A_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_B_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_B_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_A_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_A_STATUS_0 = 0x00000010
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_CILA_STATUS_0 = 0x00040041
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_B_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_B_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_CILB_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_CSI_READONLY_STATUS_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_CONTROL_0 = 0x43445300
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_COUNTER_0_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_COUNTER_1_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_COUNTER_2_0 = 0x00000000
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: *****************************************
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: Error: waitCsiFrameStart Something went wrong with waiting on frame start
Feb 11 10:51:50 maverick-nano nvargus-daemon[23005]: PowerServiceCore:handleRequests: timePassed = 1507
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: SCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 561)
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: Error: waitCsiFrameStart timeout guid 1
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: ************VI/CSI Debug Registers**********
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: VI_CFG_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: VI_CFG_INTERRUPT_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: VI_CSI_0_ERROR_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: VI_CSI_0_ERROR_INT_MASK_0 = 0x0000001f
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: VI_CSI_1_ERROR_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: VI_CSI_1_ERROR_INT_MASK_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_A_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_A_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_B_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_PIXEL_PARSER_B_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_A_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_A_STATUS_0 = 0x00000010
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_CILA_STATUS_0 = 0x00040041
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_B_INTERRUPT_MASK_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_CIL_B_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_CILB_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_CSI_READONLY_STATUS_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_CONTROL_0 = 0x43445300
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_COUNTER_0_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_COUNTER_1_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: CSI_DEBUG_COUNTER_2_0 = 0x00000000
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: *****************************************
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: SCF: Error Timeout: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
Feb 11 10:51:51 maverick-nano nvargus-daemon[23005]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)

@fnoop
Copy link
Member Author

fnoop commented Feb 11, 2020

@fnoop
Copy link
Member Author

fnoop commented Feb 11, 2020

aargh nvidia sucks, why can't they just make something that works?

Feb 11 11:12:58 maverick-nano python3[28100]: (Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 215)
Feb 11 11:12:58 maverick-nano python3[28100]: (Argus) Error Timeout: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 102)
Feb 11 11:12:58 maverick-nano python3[28100]: Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:526 Failed to create CameraProvider
Feb 11 11:12:58 maverick-nano python3[28100]: 2020-02-11 11:12:58,754 - Visiond - INFO - Creating RTSP factory element: <GstRtsp.RTSPUrl object at 0x7fa46cf600 (GstRTSPUrl at 0x7f98001a00)>
Feb 11 11:12:58 maverick-nano python3[28100]: gst_ghost_pad_new: assertion '!gst_pad_is_linked (target)' failed
Feb 11 11:12:58 maverick-nano python3[28100]: gst_pad_set_active: assertion 'GST_IS_PAD (pad)' failed
Feb 11 11:12:58 maverick-nano python3[28100]: gst_element_add_pad: assertion 'GST_IS_PAD (pad)' failed
Feb 11 11:12:58 maverick-nano python3[28100]: gst_rtsp_stream_new: assertion 'GST_IS_PAD (pad)' failed
Feb 11 11:12:58 maverick-nano python3[28100]: gst_rtsp_stream_set_multicast_iface: assertion 'GST_IS_RTSP_STREAM (stream)' failed
Feb 11 11:12:58 maverick-nano python3[28100]: gst_rtsp_stream_set_profiles: assertion 'GST_IS_RTSP_STREAM (stream)' failed
Feb 11 11:12:58 maverick-nano python3[28100]: gst_rtsp_stream_set_protocols: assertion 'GST_IS_RTSP_STREAM (stream)' failed
Feb 11 11:12:59 maverick-nano systemd[1]: maverick-visiond.service: Main process exited, code=killed, status=11/SEGV
Feb 11 11:12:59 maverick-nano systemd[1]: maverick-visiond.service: Failed with result 'signal'.

fnoop added a commit that referenced this issue Mar 4, 2020
SamuelDudley pushed a commit that referenced this issue Mar 5, 2020
@fnoop
Copy link
Member Author

fnoop commented Mar 8, 2020

fnoop added a commit that referenced this issue Mar 8, 2020
@fnoop
Copy link
Member Author

fnoop commented Mar 8, 2020

Tegra/CSI support working well now on Nano with raspberry pi camera. Only basic support/tuning in but working well with very low latency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants