Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline option to auto-restart on error #94

Closed
ivelin opened this issue Oct 17, 2019 · 3 comments
Closed

Pipeline option to auto-restart on error #94

ivelin opened this issue Oct 17, 2019 · 3 comments
Labels
good first issue Good for newcomers help wanted Extra attention is needed released
Projects
Ambianic Edge - Agile Board

Comments

@ivelin
Copy link
Collaborator

ivelin commented Oct 17, 2019

Why:
Currently pipelines execute until the source element detects end of input stream (EOS). This normally works fine for single image as well as live video pipelines. However when there is a power outage or some other interruption of the live streams, the pipelines end, which is not the expected behaviour by the user.

How:
In order to recover continuous pipelines from temporary source unavailability, we can introduce a source configuration option or a pipeline level option to restart automatically after a period of time unless the user explicitly requests the pipeline to stop.
@ivelin ivelin added good first issue Good for newcomers help wanted Extra attention is needed labels Oct 17, 2019
@ivelin ivelin added this to To do in Ambianic Edge - Agile Board via automation Oct 17, 2019
@ivelin ivelin changed the title Pipeline option to restart after EOS Pipeline option to auto-restart on error Oct 17, 2019
@ivelin
Copy link
Collaborator Author

ivelin commented Oct 17, 2019

Example error messages from the log:

0:28:27.563781095    32  0x1abbb80 WARN                 rtspsrc gstrtspsrc.c:5995:gst_rtspsrc_try_send:<source> send interrupted
0:28:27.563998424    32  0x1abbb80 WARN                 rtspsrc gstrtspsrc.c:7669:gst_rtspsrc_close:<source> TEARDOWN interrupted
0:28:27.578732810    32  0x1abbaf0 ERROR                default gstrtspconnection.c:1006:gst_rtsp_connection_connect_with_response: failed to connect: Operation was cancelled
0:28:27.578920862    32  0x1abbaf0 ERROR                rtspsrc gstrtspsrc.c:4698:gst_rtsp_conninfo_connect:<source> Could not connect to server. (Generic error)
0:28:27.579806068    32  0x1abbaf0 WARN                 rtspsrc gstrtspsrc.c:7465:gst_rtspsrc_retrieve_sdp:<source> error: Failed to connect. (Generic error)
0:28:27.581660813    32  0x1abbaf0 WARN                 rtspsrc gstrtspsrc.c:7544:gst_rtspsrc_open:<source> can't get sdp
Exiting GST process

@ivelin ivelin moved this from To do to In progress in Ambianic Edge - Agile Board Nov 8, 2019
@ivelin
Copy link
Collaborator Author

ivelin commented Nov 8, 2019

Introduced source parameter

live: true

example

sources:
   front_door_camera: &src_front_door_cam
     uri: *secret_uri_front_door_camera
     type: video
     live: true

It indicates that the source should be treaded as a live stream. Even if there are errors that interrupt reading from the source, the pipeline source should keep trying to reconnect.

@ivelin ivelin closed this as completed in 8dd4fad Nov 8, 2019
Ambianic Edge - Agile Board automation moved this from In progress to Done Nov 8, 2019
ivelin pushed a commit that referenced this issue Nov 11, 2019
@semantic-release-bot
chore(release): 0.1.0 [skip ci]
d9af692 
# [0.1.0](v0.0.4...v0.1.0) (2019-11-11)

### Bug Fixes

* add shebang at the beginning of shell script per CodeFactor recommendation ([4b2382e](4b2382e))
* adjust debug levels in log statements ([3b44bff](3b44bff))
* avsource element init params ([98c92f9](98c92f9))
* build and docs ([704660c](704660c))
* build and docs fixes ([5497da7](5497da7))
* build fix for arm docker image tag ([326c347](326c347))
* build fix for ARM Travis script crash ([cd721c9](cd721c9))
* code cleanup as suggested by codefactor ([9c19d3e](9c19d3e))
* code cleanup as suggested by codefactor ([91a110f](91a110f))
* config file cleanup for safe start of new installs ([0ed36e2](0ed36e2))
* default config uses bundled test video ([37f1d83](37f1d83))
* docker scripts ([a918c62](a918c62))
* docs and build fixes ([c440f24](c440f24))
* flaskr data dir reference when config not available ([8f8139d](8f8139d))
* graceful fps logging fallback when pipeline context is not populated ([f4cbdae](f4cbdae))
* increased test timeout to fit RPI CPU constraints ([ea34e6b](ea34e6b))
* init parameter bugs ([03f18db](03f18db))
* merge branch 'master' into dev ([428f3a3](428f3a3))
* merge branch 'master' into dev ([c6c4ec8](c6c4ec8))
* merge branch 'master' into dev ([9d8ebb2](9d8ebb2))
* merge branch 'master' into dev ([90638bc](90638bc))
* Merge branch 'master' of https://github.com/ambianic/ambianic-core ([2202c45](2202c45))
* method arguments ([e569465](e569465))
* move detection box overlay to browser, close [#123](#123) ([447d0a4](447d0a4))
* push origin masterMerge branch 'dev' ([d2dec56](d2dec56))
* reconnect on EOS when live media source ([12954b9](12954b9))
* run script chmod +x ([d3eb03f](d3eb03f))
* samples rest api ([bb7c730](bb7c730))
* sem release script ([93ed217](93ed217))
* semantic release script ([3de4c29](3de4c29))
* semantic release script ([5d33dd6](5d33dd6))
* semantic release script fix ([2765dd0](2765dd0))
* semantic release script, update node to latest stable ([164014c](164014c))
* **build:** multi arch script for docker ([82d1682](82d1682))

### Features

* **api:** add pagination to timeline API, close [#118](#118) ([40993d6](40993d6))
* close [#73](#73) Implement rolling log files ([cc1129f](cc1129f))
* close [#89](#89) Default file locations for saved inference samples ([bb035eb](bb035eb))
* close [#94](#94) source live parameter to keep reconnecting to live streams ([8dd4fad](8dd4fad))
* config file update ([b5c6851](b5c6851))
* handle crashed native processes; graceful healing; close [#125](#125) ([d716ecd](d716ecd))
* run ambianic within its docker image ([acf3fc6](acf3fc6))
* Save inference (detection) samples; close [#124](#124) ([0bfbc71](0bfbc71))
* **logs:** add pipeline contextual info to fps log messages ([b8ad547](b8ad547))
* timeline REST API ([e0212ba](e0212ba))
* **avsource:** add live property to avsource configuration element ([ea5385e](ea5385e))
* **avsource:** live streams reconnect until stopped ([d44a046](d44a046))
* **pipeline:** timeline of pipeline events ([779c5db](779c5db))
@ivelin
Copy link
Collaborator Author

ivelin commented Nov 11, 2019

🎉 This issue has been resolved in version 0.1.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Good for newcomers help wanted Extra attention is needed released
Projects
Ambianic Edge - Agile Board
  
Done
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ivelin