Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Branch: master
Commits on Oct 27, 2010
  1. @nseidle

    v2.3 Migrated to v10.10.10 of sdfatlib. Moved to inline RX interrupt …

    nseidle authored
    …and buffer. Improved the ability to receive a constant stream at 57600bps.
    
    27334 bytes out of 30720.
    
    Remember - we had to completely butcher HardwareSerial.cpp so a normal Arduino installation will not work.
    C:\arduino-xxxx\hardware\arduino\cores\arduino\HardwareSerial.cpp
    
    I removed the receive interrupt handler from HardwareSerial.cpp so that we can deal directly with USART_RX_vect in the main code. This allows us to return to the way OpenLog used to operate - by having one buffer split in half. Once one half fills, it is recorded to SD while the other half fills up. This dramatically decreases the time spent in function calls and SD recording, which leads to many fewer characters dropped.
    
    The change log at the top of the main code got so large I've moved it to a version controlled "Changes.txt" file.
    
    By making all these changes, I have broken many things. Ringp - I could use your help here. I apologize for stomping on so much of your work. I was not good enough to figure out how to re-link from the old function calls to the new sdfatlib setup.
    
    Backspace is broken - ringp, I saw this fix in one of your commits, but looking at the code, I don't see how it is supposed to work. Either way, we still get 0x08 when trying to backspace.
    
    Search for "//Error" in main pde
    ls is not working fully
    remove is not working
    fileInfo is not working
    
    New sdfatlib doesn't have SdFat.cpp so fileInfo doesn't work. These function calls are marked with //Error
    
    I have chosen to dis-allow deprecated functions:
    #define ALLOW_DEPRECATED_FUNCTIONS 0
    This forced some trivial changes to the SD write and new file creation function calls. I believe we've successfully migrated to the new version of sdfatlib.
    
    In the command_shell / read_line function : It may be better to pull directly from the UART buffer and use the RX interrupt. For now, we brute force it.
    
    Because of all these changes, I need to re-test power consumption. For now, I'm confident it's low enough.
    
    Testing with 512 buffer array size
    1GB @ 57600 - dropped very little out of 3 tests
    1GB @ 115200 - dropped very little out of 2 tests
    8GB @ 57600 - Formatted using the sd formater (32k byte allocation size). Dropped nothing.
    8GB @ 115200 - dropped very little, dropped none
    16GB w/ Lots of files @ 57600 - Drops the first part of the file because of start up delay?
    16GB w/ Lots of files @ 115200
    
    1024 array size (and 800) does not run
    
    Testing with 700 buffer array size
    1GB @ 57600 - 110300 out of 111000 bytes, 110300/111000,
    1GB @ 115200 - 111000/111000!, 109600/111000
    8GB @ 57600 - 109000/111000, 111000/111000!,
    8GB @ 115200 - 111000/111000!, 111000/111000!,
    16GB w/ Lots of files @ 57600 - 85120/111000, 85120/111000
    16GB w/ Lots of files @ 115200 - 56420 (but once it got going, log looks good). 56420.
    
    I am seeing long delays on cards with lots of files. In the above tests, the 16GB test card is a good example. It has 2GB worth of random files in a sub directory. After OpenLog boots, goes to '12<'. After I send ~500 characters OpenLog freezes for a couple seconds, then returns to normal, very fast, operation. During that down time, I believe sdfatlib is searching for an open cluster. The odd thing is that after the cluster is established (after the initial down time) OpenLog performs excellently. I am hoping to create a faux file or pre-write and save to the file or some how get this allocation done before we report the '12<' indicating we are ready. That way, if a card does fill up, as long as the host system is checking for '<', it doesn't matter how long it takes sdfatlib to find the next free cluster.
    
    You can see that we drop 700 bytes at a time. That's a bit odd - I'd expect to drop half or 350 at a time. What happens if we shrink the array size to say 256? To be expected, this is bad - way more instances of dropped characters.
    
    Added blink for each test to the OpenLog_Test sketch so we can see as the test progresses.
    
    http://www.sdcard.org/consumers/formatter/ is the site for SD card formatting. It looks like this program takes a guess at the correct block size. This could help a lot in the future.
Commits on Oct 3, 2010
  1. Fixed "888 88" bug when hitting backspace. Resolved ambigous "rm" com…

    Paul Ring authored
    …mand, new "rm" implementation. New commands: pwd (print working directory), echo <on>|<off>. Fixed bug when using "size", leaking file handle. Removed too_many_arguments_error() in order to free up memory.
Commits on Sep 30, 2010
  1. Added "rm -rf" command that will remove all contents of the directory…

    Paul Ring authored
    … including subdirectories recursively. The read-only attribute for files will be ignored.
  2. Merge branch 'master' of git://github.com/nseidle/OpenLog

    Paul Ring authored
    Conflicts:
    	OpenLog_v2/OpenLog_v2.pde
Commits on Sep 29, 2010
  1. @nseidle

    v2.21 ringp fork brought in. Wildcard remove and list commands now wo…

    nseidle authored
    …rk. Remove directory now works! Change directory up/down the tree works again.
    
    28440 bytes used of 30720.
    
    ringp brought back many great commands! Thanks ringp!
    rm LOG*.* works
    ls *.TXT works
    cd .. works again
    ls now correctly shows directories first and files following the directories.
    
    To remove a directory, you have to navigate into that directory. For example:
     >cd TEMP (you are now in TEMP directory)
     >rm -f TEMP (you are now removing TEMP, and you will be pushed back down one level of the tree)
     >ls (shows files and directories where TEMP directory used to be, but TEMP directory should be gone)
    
    ringp added new commands:
    efcount: gives the number of files in the current directory. Example: "efcount" returns "count|3". There are 3 files in the current directory.
    
    efinfo <spot>: gives info about the file in <spot>. Example: "efinfo 2" reports "LOG00588.TXT|45". File number 2 is named LOG00588.TXT and is 45 bytes in size.
    
    verbose <"on"|"off">: sets whether command errors are verbose (long winded) or just the "!" character. Example: "verbose off" sets verbose off. Then if a command like "blah" is received, then only "!>" is seen from the command line interface. This makes it easier for embedded systems to recognize there was an error. This setting is not recorded to EEPROM.
    
    Updated README.
  2. @nseidle

    Added support for removing an empty folder using "rm -f" command. Lis…

    Paul Ring authored nseidle committed
    …ting the contents of a folder lists the subfolders first and then the files in the folder.
  3. @nseidle

    Added support for wilcard listing and removal of files. This should n…

    Paul Ring authored nseidle committed
    …ow work the same as in previous OpenLog versions. "cd.." now works up to a depth of 15 folders. The depth can be increase be redefining a global variable. Added new commands "efcount", "efinfo", "eem", "verbose" that are simplifying embedded usage with OpenLog. Also fixed some code indentation problems and removed old commented code not being used any more.
Commits on Sep 26, 2010
  1. Added support for removing an empty folder using "rm -f" command. Lis…

    Paul Ring authored
    …ting the contents of a folder lists the subfolders first and then the files in the folder.
Commits on Sep 25, 2010
  1. Added support for wilcard listing and removal of files. This should n…

    Paul Ring authored
    …ow work the same as in previous OpenLog versions. "cd.." now works up to a depth of 15 folders. The depth can be increase be redefining a global variable. Added new commands "efcount", "efinfo", "eem", "verbose" that are simplifying embedded usage with OpenLog. Also fixed some code indentation problems and removed old commented code not being used any more.
Commits on Sep 21, 2010
  1. Merge branch 'master' of git://github.com/nseidle/OpenLog

    Paul Ring authored
    Conflicts:
    	Code/Makefile
    	Code/main.c
    	Code/main.hex
  2. Makefile changes. Not to be commited upstream.

    Paul Ring authored
Commits on Aug 25, 2010
  1. @nseidle

    v2.2 Modified append_file() to use a single buffer. Increased Hardwar…

    nseidle authored
    …eSerial.cpp buffer to 512 bytes.
    
    More testing at 57600. Record times look to be 2, 5, and zero milliseconds when doing a record. This means that the split buffer doesn't really make a difference? There are some records that take 150ms, 14ms, etc. At 57600bps, that's 7200 bytes/s, 138us per byte. With a 150ms pause, that's 1,086 bytes that need to be buffered while we wait... Grrr. Too many.
    
    I re-wrote the append_file function to use only one buffer. This allows us to more quickly pull data from the hardware serial buffer. Hardware serial buffer has to be increased manually to 512. This file (hardwareserial.cpp) is stored in the Arduino directory. With testing, it seems like recording is working more solidly at 57600bps. But now I'm seeing glitches at 19200bps so more testing is required before we make this the official OpenLog release.
    
    Moved input_buffer into within the append function. Attempting to shave bytes of RAM.
Commits on Aug 18, 2010
  1. @nseidle

    v2.11 Tested with 16GB microSD. Fixed some general bugs. Lowered powe…

    nseidle authored
    …r consumption.
    
    26136 bytes out of 30720
    
    Fixed issue 30. I added printing a period ('.') for non-visible ASCII characters during a 'read' command. This cleans up the output a bit. HEX printing is still available.
    
    Fixed issue 34. When issuing a long command such as "read log00056.txt 100 200 2" (read from spot 100 to spot 200 and print in HEX), the command shell would die at 24 spots. I increased both the general_buffer and 'buffer' in the command shell from 24 to 30. The limit is now 30 characters, so "read log00056.txt 100 20000 2" is allowed.
    
    Works with a 16GB microSD card! High volume test: loaded 16GB card with 5GB of data. Basic serial tests work. When running at 57600, there is an odd delay. I think it has to do with the file system doing an initial scan for an available cluster. Takes 2-3 seconds before OpenLog returns to normal. This can cause significant data loss.
    
    Fixing power management in v2. Power down after no characters for 3 seconds now works. Unit drops to 2.35mA in sleep. 7.88mA in sitting RX mode (awake but not receiving anything). There is still a weird bug where the unit comes on at 30mA. After sleep, it comes back at the correct 7-8mA. Is something not getting shut off?
  2. @nseidle

    v2.11 Tested with 16GB microSD. Fixed some general bugs. Lowered powe…

    nseidle authored
    …r consumption.
    
    Fixed issue 30. I added printing a period ('.') for non-visible ASCII characters during a 'read' command. This cleans up the output a bit. HEX printing is still available.
    
    Fixed issue 34. When issuing a long command such as "read log00056.txt 100 200 2" (read from spot 100 to spot 200 and print in HEX), the command shell would die at 24 spots. I increased both the general_buffer and 'buffer' in the command shell from 24 to 30. The limit is now 30 characters, so "read log00056.txt 100 20000 2" is allowed.
    
    Works with a 16GB microSD card! High volume test: loaded 16GB card with 5GB of data. Basic serial tests work. When running at 57600, there is an odd delay. I think it has to do with the file system doing an initial scan for an available cluster. Takes 2-3 seconds before OpenLog returns to normal. This can cause significant data loss.
    
    Fixing power management in v2. Power down after no characters for 3 seconds now works. Unit drops to 2.35mA in sleep. 7.88mA in sitting RX mode (awake but not receiving anything). There is still a weird bug where the unit comes on at 30mA. After sleep, it comes back at the correct 7-8mA. Is something not getting shut off?
Commits on Aug 4, 2010
  1. @nseidle

    v2.1 - Fixed issue 35. Dropping characters at 57600bps. Power save no…

    nseidle authored
    …t working.
    
    26058 bytes out of 30720
    Fixed a bug found by Salient (Issue 35). User settings where declared at chars which allowed them to be signed. If a user went from old firmware, to v2, the safety checks would fail because the settings would be read at -1 instead of 255. Declaring user settings as byte fixed issue.
    
    Added "a) Clear user settings" to set menu. This allows us to completely wipe an OpenLog (user settings and config file) to see how it will respond to future firmware changes.
    
    Improved the file 'size' command.
    
    Sequential logging is tested and works.
    
    Receive testing: Using the Test_Sketch found on Github, I am testing the receive reliability at different UART speeds. We need to test a lot of received data. At 57600, 115200, and both from an Arduino (lots of time in between characters becuase of software overhead) and from a raw serial port (almost no time in between characters). I am hoping to make sdfatlib hiccup at 115200, full speed, across a 1MB file. If  I can make it fail, then we can start to increase the buffer size and look at how much RAM sdfatlib has left open for the buffer.
    
    9600bps from Arduino works fine
    57600bps from Arduino drops characters
    115200 from Arduino drops characters
    
    It seems that sdfatlib takes longer to write to the SD card than the original file system from Robert Reigel. I'm thinking perhaps we should create a version of OpenLog firmware that is just sequantial logging, no fancy system menus... It might open up some RAM.
    
    If only we could get control of the UART from Arduino's clutches, we could probably handle the ring buffer much better. Not sure how to handle UART interrupts without tearing out HardwareSerial.cpp.
    
    Added README to the Test sketch. Added 115200bps to test sketch.
Commits on Jul 28, 2010
  1. @nseidle

    v2.0 - 25986 bytes out of 30720

    nseidle authored
    Welcome to version 2! We've moved from Roland Riegel's FAT library to Bill Greiman's sdfatlib. OpenLog now works with SD cards up to 16GB (that is the current largest microSD card we can get our hands on). OpenLog automatically detects and works with FAT16/FAT32 file systems. It also automatically works with normal SD cards as well as SDHC cards.
    
    Almost all the previous commands have been ported from v1 to v2. The current commands that do not work:
    cd.. - does not work. You can change to an upper directory, but you cannot navigate back down the tree.
    cat - this command was deprecated. HEX printing is now done with the 'read' command. We've added a 5th argument to select between ASCII and HEX printing.
    Wild cards do not yet work. So rm and ls do not have wild cards enabled - yet. Help us out!
  2. v2.0 - 25986 bytes out of 30720

    unknown authored
    Welcome to version 2! We've moved from Roland Riegel's FAT library to Bill Greiman's sdfatlib. OpenLog now works with SD cards up to 16GB (that is the current largest microSD card we can get our hands on). OpenLog automatically detects and works with FAT16/FAT32 file systems. It also automatically works with normal SD cards as well as SDHC cards.
    
    Almost all the previous commands have been ported from v1 to v2. The current commands that do not work:
    cd.. - does not work. You can change to an upper directory, but you cannot navigate back down the tree.
    cat - this command was depricated. HEX printing is now done with the 'read' command. We've added a 5th argument to select between ASCII and HEX printing.
    Wild cards do not yet work. So rm and ls do not have wild cards enabled - yet.
Commits on Jun 7, 2010
  1. Fixed timing issue in OpenLog.java. Removed unnecessary delay in open…

    Paul Ring authored
    …LogExecute().
Commits on Jun 3, 2010
  1. Removed unneeded #defines and fixed compiling errors when undefining …

    Paul Ring authored
    …INCLUDE_SIMPLE_EMBEDDED. Fixed timing problems in OpenLog.java driver. Smaller chages in OpenLogTest.java.
  2. Fix for disabling the logging of escape character ASCII 26 (Ctrl+z). …

    Paul Ring authored
    …When appending to a file the escape character currently used will also be appended to the file. In order to avoid this problem issue the "binary off" command at the terminal.
  3. Fixed issue 30 (Character Ctrl+Z on OpenLog). Escape characters (in t…

    Paul Ring authored
    …his case Ctrl+Z) are no longer appended to the file. Fixed some other small terminal endline problems like for example 2 newlines displayed after each other when running terminal commands. Small indentation fixes.
Commits on Jun 1, 2010
  1. Support for using OpenLog with Parallax Javelin Stamp. Simplified set…

    Paul Ring authored
    … of commands -that can be used with any other micro processor- to count files in a folder (efcount), get file info (efinfo), turn off echoing (echo <on/off>), turn off extra information when something goes wrong (verbose <on/off>) and also a special command to ease the communication and separate the actual data from the result of a command (eem <on/off>). Created a "Drivers" folder to host drivers for other microcontrollers. Corrected some other minor bugs, like write command not working properly when using offset.
Commits on Apr 28, 2010
  1. @nseidle

    v1.61 Small PCB change. Fixed version # in help menu.

    nseidle authored
    Fixed the firmware version in the help menu to v1.61.
    
    Updated Eagle files to Eagle v5.9. Fixed weird airwire. Changed D1 LED from Green to Blue. Will only affect new production after 4-28-10.
    
    Closed some tickets and wrote some more example Arduino code:
    http://forum.sparkfun.com/viewtopic.php?t=21438
Commits on Apr 21, 2010
  1. @nseidle

    v1.6 Adding config file. OpenLog can now be configured either through

    nseidle authored
    CONFIG.txt or through system menus.
    
    What happens if I reset the system by pulling RX low, but the config file
    has corrupt values in it? If config file has corrupt values in it, system
    will default to known values 9600/ctrl+z/3/newlog. If config file is
    empty, system resets to known values  After some massive testing, and lots
    of code to check for illegal states, it looks to be pretty stable. The
    only problem is that we're running out of RAM. The buffer had to be
    decreased from 900 bytes to 700 bytes to facilitate all the config file
    checking. Testing at 57600bps, unit runs very well over 40kb test file on
    straight RS232 connection. That's pretty good. Testing at 115200 on
    straight connection, unit will drop a buffer every once and a while. Not
    great, but not much we can do if the SD card times out for ~150ms while
    it's writing.
    
    8 bits to the byte plus a start/stop bit = 10 bits per byte
    @ 9600bps = 960 bytes per second. Buffer will last for 729ms
    @ 57600bps = 5760 bytes per second. Buffer will last for 121ms
    @ 115200bps = 11520 bytes per second. Buffer will last for 60.7ms
    
    So if the SD card pauses for more than 60ms, 115200 will have lost data,
    sometimes. All other baud rates should be covered for the most part.
    
    SD cards with larges amounts of data will have increased pause rates.
    Always use a clean card where possible.
  2. v1.6 Adding config file. OpenLog can now be configured either through…

    Nathan Seidle authored
    … CONFIG.txt or through system menus.
    
    What happens if I reset the system by pulling RX low, but the config file has corrupt values in it? If config file has corrupt values in it, system will default to known values 9600/ctrl+z/3/newlog. If config file is empty, system resets to known values	After some massive testing, and lots of code to check for illegal states, it looks to be pretty stable. The only problem is that we're running out of RAM. The buffer had to be decreased from 900 bytes to 700 bytes to facilitate all the config file checking. Testing at 57600bps, unit runs very well over 40kb test file on straight RS232 connection. That's pretty good. Testing at 115200 on straight connection, unit will drop a buffer every once and a while. Not great, but not much we can do if the SD card times out for ~150ms while it's writing.
    
    8 bits to the byte plus a start/stop bit = 10 bits per byte
    @ 9600bps = 960 bytes per second. Buffer will last for 729ms
    @ 57600bps = 5760 bytes per second. Buffer will last for 121ms
    @ 115200bps = 11520 bytes per second. Buffer will last for 60.7ms
    
    So if the SD card pauses for more than 60ms, 115200 will have lost data, sometimes. All other baud rates should be covered for the most part.
    
    SD cards with larges amounts of data will have increased pause rates. Always use a clean card where possible.
Commits on Apr 14, 2010
  1. @nseidle

    v1.51 check_emergency_reset, default break character is ctrl+z 3 time…

    nseidle authored
    …s, example Arduino sketch
    
    Added function from mungewell - check_emergency_reset. This has improved testing of the RX pin.	There was potential to get a false baud reset. There is still a chance but it's now much less likely.
    
    If OpenLog is attached to a Arduino, during bootloading of the Arduino, ctrl+z will most likely be sent	to the Arduino from the computer. This character will cause OpenLog to drop to command mode - probably not what we want. So I added user selectable character (ctrl+x or '$' for example) and I added user selectable number of escape characters to look for in a row (example is 1 or 2 or 3, '$$$' is a common escape sequence). The default is now ctrl+z sent 3 times in a row.
    
    Added an example Arduino sketch (from ScottH) to GitHub so that people can easily see how to send characters to	OpenLog. Not much to it, but it does allow us to test large amounts of text thrown at OpenLog at 57600bps.
Commits on Apr 7, 2010
  1. v1.5

    unknown authored
    Added 4800bps and 19200bps support
    
    Added power saving features. Current consumption at 5V is now:
    In default append mode:
    	6.6/5.5mA while receiving characters (LED On/Off)
    	2.1mA during idle
    In command mode: 3.2/2.1mA (LED On/Off)
    
    So if you're actively throwing characters at the logger, it will be ~6mA. If you send the logger characters then delay 5-10 seconds, current will be ~2.5mA. (Unit records the characters in the buffer	and goes into idle more if no characters are received after 5 seconds)
    
    These power savings required some significant changes to uart.c / uart_getc()
Commits on Mar 31, 2010
  1. OpenLog v1.4 Compiled HEX

    unknown authored
  2. v1.4

    unknown authored
    Eliminates errors while logging at 57600bps.
    Added exit options to the two menus (set and baud)
    Also added display of current settin to two menus (Ex: "Baud currently: 57600bps")
    
    Added '!' infront of 'error opening'. This pops up if there is an error while trying to append to a freshly created new log (ex: LOG00548.txt is created, then errors out because it cannot append). '!' was added so that user can parse against it.
    
    Replicated logging errors at 57600 using 5V Arduino
    Unit would systematically glitch during logging of 111054 bytes
    
    Increasing buffer to 1000 characters caused URU error.
    URU: Unit Resets Unexpectedly
    
    To recreate URU error. Type "append ". Include the space. If you get "!error opening", then things are fine. If you get "!error opening#" where # is a weird character, then type 'ls' and the unit will unexpectedly reset (URU error). I believe this is attributed to a memory overrun somewhere in the FAT system.
    
    Changed buffer size to 900 and declared the character buffer as volatile
    #define BUFF_LEN 900
    volatile char input_buffer[BUFF_LEN];
    
    This increase to the buffer allows for clean logging of 444055 bytes with no URU errors.
    
    Experimenting with Scott's SD cards (customer gave cards on loan for recreating logging errors):
    Card with single ~740mb file produces errors when trying to open/append to new log.
    Card with less stuff on it logs full 444055 bytes correctly.
  3. Adding compiled HEX of latest version to repository (per bkuhns reque…

    unknown authored
    …st) so that the user doesn't have to have a full WinAVR installation (or equivalent) to get the latest firmware.
Commits on Mar 24, 2010
  1. v1.3

    unknown authored
    Added sd_raw_sync() inside append_file. I believe this was why tz1's addition of the timeout buffer update feature was not working. Auto buffer update now working. So if you don't send anything to OpenLog for 5 seconds,	the buffer will automatically record/update.
  2. Corrected version number in system menu, removed EEPROM log # testing…

    unknown authored
    … function, verified ls LOG*.* function from ringp.
  3. Testing and adding loop catching.

    unknown authored
    Test log roll over error 65533 to 65536: Doesn't work. Unit gets caught in endless loop.
    
    So I added two sections to newlog() that check for max value of 65534 (65535 looks like un-initiated EEPROM spots). Works well and errors out gracefully.
    
    Unit will throw error "!Too many logs:1!" if there are too many logs then gracefully drop to command mode.
    If you are using OpenLog, parse for:
    "12~<" means everything is good
    "12!...." means there is an error while attempting to open a new log.
    TODO: Use this style of error throwing for other types of errors (append errors, card errors maybe, etc).
    
    New rm LOG*.* command: Tests well, I love this added feature. Works great!
    
    LOG# increase to 16-bit (65535 max logs) looks good. The EEPROM storage locations look correctly updated. Testing works well.
Commits on Mar 23, 2010
  1. Using "init" command to initialize the SD card will not reset file sy…

    Paul Ring authored
    …stem handles stored in global arrays in fat.c. Calling "init" several times will cause other commands to stop working, like "cd <folder>" or "ls". This problem has now been corrected.
Commits on Mar 20, 2010
  1. Wildcard removal and listing of files. Changed version number to non-…

    Paul Ring authored
    …official version. Fixed other small code indentation problems.
Something went wrong with that request. Please try again.