Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Pattern in data dropping #111

Closed
gemmer opened this Issue Apr 10, 2012 · 3 comments

Comments

Projects
None yet
3 participants

gemmer commented Apr 10, 2012

Hi Nathan,

Currently, I've got a setup to record six-axis IMU data at 200Hz to the OpenLog, and pump it through a Bluetooth Mate at the same time. Each register of the IMU is two bytes of information - along with a two-byte status report, an indexing variable, and an end-of-frame character, it's 18 8-bytes per IMU access, or 28800 bits/s.
So, I know that I'm already playing with fire, since the margin of absolute safety is writing at 9600baud, and minimum SD spec is a baud rate of 16400 (assuming 512 bytes written at 250ms intervals). Accordingly, I picked up some SD cards from Amazon - 16GB, class 10, which should be up to the task. (class 10 nominally is 10MB/s, and I'm producing ~10MB/hr).

Well anyway, there are gaps in the logs, indicated by when two adjacent frames of data don't have adjacent indices. The size of the missing data is what I'd expect - 28 and 4/9 of a frame missing, 512 bytes.
What is puzzling me though is the pattern of how the data's dropped. I'm attaching two figures that show data discontinuities - the horizontal is sequential frames, left-to-right, and the vertical is sequential 2000-frame blocks, top-to-bottom, so a point x,y from the top left of the graph is the (x + y*2000)th frame of data.
It's that they seem to occur in pairs, and sometimes there's a break of recorded frames in between those pairs, but not a full 512 byte break. Data is being dropped 512 bytes at a time, but it's not being recorded in 512 byte blocks.
Why is the data loss happening in pairs? Is the data meant to be recorded in 512 byte blocks, and if so, why wouldn't it?

Speaking briefly about my recording setup, the data is getting shunted to the OpenLog as soon as it's received from the IMU, so there's nine sixteen-bit write commands, and about five ms of reflection time before the next. Is it better for the OpenLog is I fed it the characters in larger strings?

PICTURES
(http://img152.imageshack.us/img152/6245/redau.png) Card one's missing data (marked in black)
(http://img406.imageshack.us/img406/4142/greenk.png) Card two's missing data (marked in black)

aspineux commented May 8, 2012

The pattern is maybe related to the filesystem handling. At regular interval (vevery X byte) the FS must allocate new structure and save the ones that are full.

Owner

nseidle commented Jan 15, 2013

Thank you for clearly capturing the issue! The graphs help a ton. I'm trying to replicate your issue but I'm afraid the codebase has improved since you were experiencing problems. Are you still running into this issue?

Owner

nseidle commented Apr 8, 2013

No activity from reporter. Closing.

@nseidle nseidle closed this Apr 8, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment