timelapse ends before print ends (rasperry pi)

[LangBalthazar]

I truly love OctoPrint!

I am using the latest linked image image and have done an update for master and then moved to devel with same results.

The timelapse video stops after the first couple of layers.

I have tried switching resolution from 1280 to 640 and frame rate from 25 to 10.

I have tried Zchange and timed on 30s, 2s, 1s

I have tried with camera directly to rpi and via a 2.5A usb hub. same result

But still the timelapse video cuts out after a few layers (not sure if it is the same amount of layers every time but i could be)
example of timed, the print was a 10hour succesfull print
the file was 49.0MB
http://youtu.be/jKv7U1dg9hw

30min bolt print that had a short timelapse
the file was 2.8MB
http://youtu.be/X8PsTNHz48c

I have so far done 11 prints with timelapse and all have been cut before the print was done

Could it be the Trust webcam that sucks or is there something that might help in the settings?

Any ideas on how to test what is causing this_

Best regards
Balthazar Lang

[LangBalthazar]

Found the problem:

If someone downloads the timelapse video file before it is completely "done"/ moved / rendered the video ends right there when someone tried to download it.
(not sure what happens after the print is done but the size of the file increases slowly after the print is finished).
Working timelapse but ruind print http://youtu.be/SvmHZGhrJdA

Could the download option be made inactive until the file is completely finished?

[foosel]

I'm a bit surprised by that tbh, wouldn't have expected reading access to a file being rendered to be a problem...

[LangBalthazar]

You are right, it is not the problem i was so happy i got i full timelapse i jumped the gun.

:( i just hade another failed timelapse for a 3hour print.

So i have no idea what is causing it.

any tips on what i could monitor or test?

[foosel]

It might be interesting to know if there are any error messages in the log
file (~/.octoprint/logs/octoprint.log). Also take a look into
~/.octoprint/timelapse/tmp, are the images in there in order or are any
numbers missing?

[LangBalthazar]

Hi,

I just did a 1:30 hour print and got 709 sequential files (noting missing in the series) ending with "tmp_00708.jpg" same as before it stops after the first layers.
cp to a folder that is web accesible: http://83.233.18.254/static/tmp_00708.jpg
The timelapse was timed with 1sec so roughly 11min in.

The log "~/.octoprint/logs/octoprint.log" from today. No errors today
2014-04-01 08:03:39,593 - octoprint.gcodefiles - INFO - Migrating metadata if necessary...
2014-04-01 08:03:39,632 - octoprint.gcodefiles - INFO - Updated 0 sets of metadata to new format
2014-04-01 08:03:39,954 - octoprint.server - INFO - Listening on http://0.0.0.0:5000
2014-04-01 08:09:29,862 - octoprint.server.util - INFO - New connection from client: 10.0.0.4
2014-04-01 08:15:13,237 - octoprint.server.util - INFO - Client connection closed
2014-04-01 08:15:18,179 - octoprint.server.util - INFO - New connection from client: 10.0.0.4
2014-04-01 08:41:02,546 - octoprint.server.util - INFO - Client connection closed
2014-04-01 08:41:09,053 - octoprint.server.util - INFO - New connection from client: 10.0.0.4
2014-04-01 08:41:21,548 - octoprint.server.util - INFO - Client connection closed
2014-04-01 08:41:28,618 - octoprint.server.util - INFO - New connection from client: 10.0.0.4
2014-04-01 09:11:15,249 - octoprint.server.util - INFO - Client connection closed
2014-04-01 09:11:21,623 - octoprint.server.util - INFO - New connection from client: 10.0.0.4
2014-04-01 10:48:45,207 - octoprint.server.util - INFO - Client connection closed
2014-04-01 10:48:51,422 - octoprint.server.util - INFO - New connection from client: 10.0.0.4

I also change power supply today to a 2A apple original iPad charger. Previously 1A Apple iPhone charger. But same results as above.

Is there some way to monitor if the connection to the webcam is lost. It seems that webcam (usb) is lost during the print. Because my preview/stream under "control" also stopped working after the print. Probably has been this way also for my previous prints.

Or monitor if the mjpeg script crashes for some reason since OctoPrint-"the printing part" is running smoothly only the images stop coming in.

If i look at top -c now after the print everything looks normal noting is currently eating resources.

ideas?

[LangBalthazar]

Did a new print run with some changes;

1. mild overclock 800mhz...
2. memorysplit 16mb for GPU (figurerd that graphics is not necessary for a webserver)

and this time i got an error in the log:
2014-04-01 14:13:27,189 - octoprint.server.api - INFO - Performing command: sudo shutdown -r now
2014-04-01 14:14:11,767 - octoprint.gcodefiles - INFO - Migrating metadata if necessary...
2014-04-01 14:14:11,803 - octoprint.gcodefiles - INFO - Updated 0 sets of metadata to new format
2014-04-01 14:14:11,975 - octoprint.server - INFO - Listening on http://0.0.0.0:5000
2014-04-01 14:30:18,500 - octoprint.server.util - INFO - New connection from client: 10.0.0.4
2014-04-01 14:30:50,281 - octoprint.server.util - INFO - Client connection closed
2014-04-01 15:57:41,942 - octoprint.server.util - INFO - New connection from client: 10.0.0.10
2014-04-01 15:57:50,465 - octoprint.server.util - INFO - Client connection closed
2014-04-01 16:39:07,045 - octoprint.server.util - INFO - New connection from client: 10.0.0.4
2014-04-01 16:40:06,022 - octoprint.server - CRITICAL - Now that is embarrassing... Something really really went wrong here. Please report this including the stacktrace below in OctoPrint'$
2014-04-01 16:40:06,082 - octoprint.server - ERROR - Stacktrace follows:
Traceback (most recent call last):
File "/home/pi/OctoPrint/src/octoprint/server/init.py", line 214, in run
IOLoop.instance().start()
File "/home/pi/oprint/local/lib/python2.7/site-packages/tornado/ioloop.py", line 627, in start
event_pairs = self._impl.poll(poll_timeout)
MemoryError

http://pastebin.com/q0m066wQ

I do not think this was related to my short timelaspe issue since the server crashed when i tried to upload a new gcode file after a successful print with failed timelapse.

[foosel]

That last one rather looks like for whatever reason the server ran out of memory while you tried to upload the file -- how big was it?

Regarding your timelapse issue, please try the following.

1. ssh into your pi
2. cd ~/.octoprint/timelapse/tmp
3. avconv -i tmp_%05d.jpg -vcodec mpeg2video -pix_fmt yuv420p -r 25 -y -b:v 10000k -f vob test.mpg (I hope I got the parameters correctly...)

This should render the movie locally directly in the shell. Might take a while, but should give you output regarding what it does. The whole output of the command might be interesting, so please capture that (=> pastebin or gist).

For comparison, I get something like that for a quick test timelapse on my devel machine:

ffmpeg version N-44818-g13f0cd6 Copyright (c) 2000-2012 the FFmpeg developers
  built on Sep 27 2012 19:30:20 with gcc 4.7.1 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-pthreads --enable-runtime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --enable-libass     --enable-libcelt --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libnut --enable-    libopenjpeg --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libutvideo --enable-libvo-aacenc --enable-libvo-amrwbenc     --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      51. 73.101 / 51. 73.101
  libavcodec     54. 59.100 / 54. 59.100
  libavformat    54. 29.104 / 54. 29.104
  libavdevice    54.  2.101 / 54.  2.101
  libavfilter     3. 17.100 /  3. 17.100
  libswscale      2.  1.101 /  2.  1.101
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
Input #0, image2, from 'C:\Users\Gina\AppData\Roaming\OctoPrint\timelapse\tmp\tmp_%05d.jpg':
  Duration: 00:00:01.84, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj420p, 640x480 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 25 tbn, 25 tbc
[vob @ 044570e0] VBV buffer size not set, muxing may fail
Output #0, vob, to 'C:\Users\Gina\AppData\Roaming\OctoPrint\timelapse\20mm-box.0.05mm_20140401192856.mpg':
  Metadata:
    encoder         : Lavf54.29.104
    Stream #0:0: Video: mpeg2video, yuv420p, 640x480 [SAR 1:1 DAR 4:3], q=2-31, 10000 kb/s, 90k tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg -> mpeg2video)
Press [q] to stop, [?] for help
frame=   46 fps=0.0 q=2.0 Lsize=     874kB time=00:00:01.80 bitrate=3977.7kbits/s    
video:862kB audio:0kB subtitle:0 global headers:0kB muxing overhead 1.431756%

[LangBalthazar]

Regarding the upload sitenote:
The file was 4.5mb, i have around 4.4gb free space on my sd card and i have uploaded 13mb of gcode prevoisly without any problem. I also got a bad gateway response in the ajax front end before it crashed.

I did first an ls and then your command
https://gist.github.com/LangBalthazar/9919526

the video looks like this
http://youtu.be/MmWHhjEhflc
the last image in the series looks like this
http://83.233.18.254/static/tmp_01123.jpg

the successful prints look like this

[foosel]

Hm... Looking at that I think I agree with you that it has something to do with OctoPrint not being able to capture any images from the webcam anymore after the first ~1000. I just pushed a change to the devel branch that should enable better error logging in such cases, could you pull that and retry? That of course won't solve the issue, but might help better pin-pointing where it lies.

[LangBalthazar]

I am now at Branch: devel, Commit: 1a7a468

i did 4 prints today:
no.1, got a full timelapse : https://www.youtube.com/watch?v=U851Lu8gQVM
no.2, got nothing
no.3, got nothing
(noticed that there was no stream under the control tab and did a reboot)
no.4, got a full timelaspe. https://www.youtube.com/watch?v=dmlwM4YuYi4

the last print (4) was successful and had these images and the total log so far from today lookes like this:
https://gist.github.com/LangBalthazar/9934366

So much better results but i still can not see in the log why the camera stops working.

[foosel]

Hm... I honestly don't get why there are no error log entries, there should be if capturing fails, which it obviously does because otherwise you'd have some pictures. Just to make sure that the update actually got "applied", could you make sure that date shows the correct time and restart the server? We recently had such an issue (wrong date set on the Pi) causing python to not detect new code when being run and therefore not creating new byte code, instead using the old one.

What camera are you using there btw, and is mjpg_streamer stil running when your video stream vanishes?

[LangBalthazar]

Hi,

I had an incorrect timezone so i switched to Sweden and did a new pull and got a new update:
"Branch: devel, Commit: bf9d5ef"

I also have some errors but they can be related to other hardware issues because i did not use a powered usb hub when i got them and did a couple of reboots during the time.
example: "2014-04-05 10:33:08,712 - octoprint.timelapse - ERROR - Could not capture image /home/pi/.octoprint/timelapse/tmp/tmp_00213.jpg from http://127.0.0.1:8080/?action=snapshot"
This morning log is here: https://gist.github.com/LangBalthazar/9989470

I will run a few prints now with the powered usb hub to see what happens in the logs and report back what happens.

[LangBalthazar]

Hi again,

so now i have done one failed print and one successful print this afternoon but no time lapse on either one.

I did a reboot at "2014-04-05 16:00:15,161" before my second print but still no timelpse
the log is here:
https://gist.github.com/LangBalthazar/9989470

From 11:01:13 i have had a powered usb hub but nothing in the logs about any problems but i got no new time lapse

[foosel]

Does mjpg-streamer run when you don't get a timelapse? It appears from the
logs that it doesn't/crashes, so that's not an issue with octoprint but
rather something in your setup (the webcam?) causing mjpg-streamer to die.

[LangBalthazar]

Hi,

Today i tried with a raspicam but everything went bad. I lost connection with the printer 4 times and have had no succesfull prints today.
All my prints today with a raspicam end up with:
"Printer requested line 283 but no sufficient history is available, can't resend"
and the print freezes...

But I got 1 timelaspe of a print that stops on the first line 9sec in to the print. :(

The log is here:
https://gist.github.com/LangBalthazar/10008755

I have to take a step back now and try to figure out why everything started crashing and remove the raspicam.

[LangBalthazar]

Back on track.

I have a theory that my prints i causing electrical distrubances.

With the raspicam i get a black line across the video. if i move the usb cables the line/distrubance changes.

example where the timelapse worked but with disturbance that disappear when moving cables:
https://www.youtube.com/watch?v=5ybzH1pRIK8

Since the logs never gave any answer this might the problem all along that the electrical disturbances caused the usb webcam to loos connection, maybe?

I will google for some tips on removing electrical disturbance over usb and see if this gives any better results.

I think we can close this issue for now. what do you think?

[ymilord]

I've been having this issue for a while. To the point where I pretty much stop using the timelapse function. I am not running on a RPi. I am using a UDOO Quad (udoo.org) with a Logitech 9000 Pro (http://www.logitech.com/en-gb/support/quickcam-vision-pro-9000-mac) running Debian but it is using the RPi ver. of mjpg-streamer and on the current build of the devel branch.

in my logs I get...

Ignoring empty buffer ...
Ignoring empty buffer ...
Unable to dequeue buffer: Invalid argument
Error grabbing frames

then mjpg-stream crashes. It's annoying to have anywhere from 30% to 85% of the print. Or sometimes one frame. In the beginning I would restart the daemon if I caught it. Like I said, It's gotten to the point where I don't even use it anymore.

[foosel]

@ymilord does that webcam work fine under linux in general? Because that above reads like something is seriously off there.

@LangBalthazar it might be electrical interference (at least I highly suspect the raspicam acting out being caused by this, the cable isn't really shielded at all), I'd suggest keeping the cables short, not crossing them with anything carrying large currents (e.g. motor wiring!) and trying to exchange them if that doesn't help. However, it also still might be some incompatibility between your webcam and mjpg-streamer/linux. I had similar issues with a Microsoft Lifecam Studio in the past, it just stopped working after a time and crashed the mjpg-streamer process. Switching to a different webcam (in my case a Logitech C270) solved the problem completely. And last but not least, there's always the question of power consumption. I run the webcam + wifi dongle off of a powered USB hub that also powers the Pi itself, this way the two most power hungry appliances connected to the Pi don't get fed via the USB port of the Pi but by the power supply of the hub. That has proven to make things more stable for me.

[ymilord]

Well it seems to crash only when a job is running.

[LangBalthazar]

@Fossel

I have been running with the raspicam now for 10-ish prints and the timelapse works without any problems.

My power setup looks like this:
Apple 2A PSU
super short cable
RJ45 ethernet (no wifi dongel)
only printer connected via usb

Examples of fully functional.
https://www.youtube.com/watch?v=LOlTJhlqtoM&list=PLybMZoBKaBihYmwj0jfXCK3qStWX3Q-cd

So i belive that my issue was related to electrical interferance that caused the usb camera to loose connection and not octoprint.

Thank you for all your support and this great software.

[LangBalthazar]

From my side we can close thus issue