Running code off of an EEPROM #344
Replies: 28 comments
-
Posted at 2015-01-12 by DrAzzy Now, you may be wondering what the point of the cleanup() function is... If I have a microcontroller with saved code on it which I use, I will often do something like this for onInit()
So I can enter maintenance mode by holding down that button on reboot, and in that mode, I can connect to the console, load the utility functions from the eeprom and modify the functions in it, and then call cleanup(), and then if I need to modify the normal code too, I can call save() and there won't be any cruft from the eeprom maintenance to clog up the saved code. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-12 by @allObjects Inspires me for something: put like the Apache Documents folder (hierarchy) onto an SD cart - static context, templates, (rendering) functions (like taglibs as used in .jsp in the J2EE world), and what ever is needed to serve an http request(ed html page) - then add some execution container onto Espruino, which able to load it dynamically and temporarily from the SD card and create the html response. To start simple, adding a templating framework - such as the one from _ (underscore.js) would go quite far for that. In _ the templates are actually compiled into anonymous javascript functions... (currently dealing with backbone, underscore, jQuery, and... - and got to like underscore a lot... it is quite a cool js lib for doing a lot of things... of course, also things in a browser context... but that part could be dropped...). |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-12 by @allObjects ;-) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-13 by @gfwilliams That's really neat! Just had an idea (it's pretty nasty), but if you created your own I'm going to have to implement some support for the DS2431 and similar EEPROMs. While they're not that big it'd be amazingly handy to be able to solder on a simple 2 or 3 wire device and get some nonvolatile storage. Because they don't draw much power you could just wire them between any available IO as well. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-13 by DrAzzy Oooo! Neat! I didn't know those existed - you know how I am about eeproms! I'll order some of those one-wire EEPROMs tonight and see if I can get a module written up :-) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-13 by @gfwilliams Thanks! Just did a quick look and if you check farnell for 3 wire through-hole EEPROMS you get:
So I'd say it's close between the first two. This page implies that the DS2433 it may be EOL, but I think the DS24B33 isn't. Hopefully they're very similar in terms of protocol though. Microchip do some really nice, really cheap ones, but while there are supposed to be through-hole versions available, you don't seem to be able to buy any of them :( It'd be a very handy little add-on board though. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-13 by DrAzzy Ordered a DS24B33 and DS28EC20 (20kbit). The cost per bit is... $3 for 4 kbit? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-13 by @allObjects @gordon, when creating this support, could you make it FRAM compatible? FRAMs do not need wait state(s) as EEPROMs do. See 256-Kbit (32 K × 8) Serial (SPI) F-RAM - Ferroelecric RAM as an alternative to EEPROM. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-13 by DrAzzy I thought you'd already made a module for the FRAM? As I understood it, he's just talking about adding a module for it, which I've volunteered to do for the one-wire ones. Do you have the beginnings of an FRAM module, at least? I'd be happy to reformat it as a module, but I don't have any of the chips on hand to test with. I'd like to make all the EEPROM modules behave as close to identically as possible for the user. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by @allObjects @drazzy, yes, but only on a very low level: read and write bytes at a memory location. Basically, the serial FRAM/MRAM behaves like the serial EEPROM - except there are no wait states for write. No wait states for writes and unlimited write cycles would open up new application files: some memory segments in the whole memory space could be FRAM/MRAM and would serve as persistency with the simple option of having or moving data there. For my own use I have started writing an (string) object store with garbage collect, but no file system yet. As I understand your intention is to have a file system with files on a (what ever) device that you can access serially. Correct? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by DrAzzy Oh, no, I'm not trying to make a file system or anything - this is just a system to store snippets of code on eeprom, and run them from there (as a way of saving precious espruino memory) The modules I'm referring to are the AT24 and AT25 modules, which provide the basics of interacting with those - read as bytes or string, write as bytes or string. Counting the FRAM modules, we've got 4 options for chips that do essentially the same thing (byte-addressed non-volatile storage). I figure the modules for these all ought to behave the same way, to minimize confusion. FRAM's lack of delay on writing, and unlimited write cycles do open up some new possibilities, although it's hard to wear out EEPROMs in realistic lengths of time, unless you're writing almost constantly - 100k writes is still a lot of writes... What chips are you using, anyway? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by @gfwilliams
As @drazzy says, that'd be another module. The reason I want to support these EEPROMs is that (while expensive) they're fantastically simple to wire up - needing only 2 wires connected to any 2 pins.
Yes, that'd be great. Ideally just I guess if I made At some point in the future I could make a 'fake eeprom' that did the same via inbuilt flash memory too. Perhaps then the filesystem you'd posted up earlier could be a module that worked on top of any EEPROM? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by @allObjects I understand... so we go after exactly the same problem... ;-)... The details about the FRAM chip and its feature are described at the bottom of the very first post of the thread. My problem raised from the fact that Espruino has quite limited memory. In my case I wanted to write a calibration / recalibration function for the resistive touch screen. Calibration is done by placing graphical X's in - for example 20 x 20 px - squares in all four corners and the center of the display and have the user to tap them. Based on the read resistances and the given screen size the, the mapping x/y-resistances of the touch screen to the x/y-coordinates of the display is adjusted. Before writing the X markers, the current display buffer content has to be saved and after using the X markers restored. On a ILI9341 controller - with 262K colors - each pixel uses 16 bits: 5 Red, 6 Green, 5 Blue. For 5 markers of 20 x 20 px square, 5 * 20 * 20 * 2 = 4kBytes - at least - have to be saved and restored (see post of Resistive Touchscreen directly (no touch controller) thread). It is 4kB when hving packed the color bits into tow bytes with shift/unshift/mask operations for the storing and using. Using one byte for each color makes the processing simpler, but the amount of data grows by 50% to 6kBytes. In one of my code versions I show one calibration X and square after the other to reduce the amount of bytes to 400 respective 600 bytes at a time. I didn't want to put aside 600 bytes just in case for a user wants to recalibrate the touch screen, because some of my apps in which I planned to use the touch screen, have already run out of memory without that UI component imbedded... so I was thinking about a temporary storage. Of course, there are also savings in these apps by breaking them up into modules and load them minified. Btw, this approach of creating modules is helpful regarding memory consumption reduction in all apps, with the downside of running out of Google closure compiler services, which are a part of the IDE's uploading to the board (see too many compiles performed recently. Try again later). I knew about serial EEPROMs, but knew also about the EEPOM limitations and cumbersome page writing. This made me look for some serial RAM, since I did not need the persistency aspect. Just by accident I stumbled on FRAM/MRAM technology. Even though - as just said - I did not need persistency, I liked it, because it would give me both: fast unlimited, simple RAM like read and write, and at the same time persistence, for - for example - storing any dynamic configuration... So I got myself 2 chips... of which one I fried after I just got it working :(((8888 . So I unsoldered the 1st one and soldered the 2nd one and rewired it and it working ever since. I was asking for compatibility, because the the FRAM commands include EEPROM unique commands for the possibility to replace EEPROMs w/ FRAMs without having to change the driver software. This must have been a very early though, because EEPROM's capacity and capacity by price left FRAMs/MRAMs technology in the dark - especially FRAM. For MRAM it looks a bit better, but I could only find large capacity chips with Ball packaging... even though http://www.everspin.com has also 256Kb SMT chips - to mention the smallest. Btw, there are various parallel interfaces to these chips as well as the SPI interface... with a serial speed up to impressive 40MHz - no wait states in any operation. With a FRAM chip working, I began the development of a memory manager - the (string) object store - which allows storing and retrieving strings in the serial FRAM. To not get lost in the details right from the beginning and slowed down by always pushing to the board, I developed a code reference model in the browser in javascript with visualization... which I attach to this post and you can play with in a browser. It is a single html file with embedded javascript. The nice thing is that the memory matrix and its content is displayed with a color scheme to see what is going on. In the code, the bare metal is to the left, and the visualization code is to the right about half a page indented. For now I have some happy paths, nothing optimized yet, and no error messaging... My plan is to move that later onto Espruino with the low level things into assembler/compiled functions. Basically, the memory has 3 segments:
When segment 2 and 3 touch each other - in other words - memory rans out of new space on attempt of storing an object, a garbage collect of segment 2 happens with the hope to reclaim not anymore used space and complete the store operation. If after garbage collection there is still not enough space, the storing process ran definitely out of memory. Usage example of the high-level browser app as implemented in the attached html file: Step 1. After loading you see a - (shot) S01: @ 0000: 000A - length of the descriptor in bytes Step 2. After storing (writing) the first string - ABCD - to the memory - S02: @ 0004: 0013 - addr of the 1st new free byte for object storage @ 000A..0012 - stored object containing the string ABCD and admin information Step 3. After storing (writing) a second string - efghijklmn - to the memory - S03: @ 0004: 0022 - addr of the 1st new free byte for object storage Attachements:
Attachments: |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by @allObjects Step 4. After storing (writing) a third string - 1234567890 - to the memory - S04: Step 5. After retrieving (read) of the string with id - 00FC - ('efghijklmn') from memory - S05: Step 6. After deleting (delete) of object with id - 00FC - ('efghijklmn') from memory - S06: @ 0008: 00FC - addr of last freed object pointer / object ID (begin of chain of freed object pointers) @ 0013..0021 - stored object still there, but type is overwritten with 80 to indicate deleted object Attachments: |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by @allObjects Step 7. After updating with (string) value of 678 of obj with id 00FA - S07: @ 0022..0029 - updated object @ 002A..0030 - 'new' unused object as 'built' from the not anymore used in place space @ 00FA..00FB: 0022 - object data addr at obj ptr / id - UNCHANGED Step 8. After updating with (string) value of 1234 of obj with id 00FA - S08: @ 0031..0039 - updated object @ 00FA..00FB: 0031 - new object data addr at obj ptr / id Step 9. After garbage collection - S09: All unused object space are purge by moving remaining active object next to each other, and the object data pointers are updated. Another needed feature I did not implement yet is the data type 01: keyed or named (string) objects. Because after a reset, the application has to be able to retrieve object ids in order to re-access them. This feature can be implemented in a first step by storing an array of object ids in a (string) object as the very first object, because the id of this TOC (table of content) (string) object's id is always size of the memory - 2. So much for now. Some more doc will follow / updated. Attachments: |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by @gfwilliams @drazzy - sorry, I got the parts and couldn't wait: http://forum.espruino.com/conversations/260971/ I'd be interested to see if it works for you? @allObjects - this is what I meant... So ideally you'd have two parts: One that accesses the EEPROM/FRAM/MRAM/etc, and another that handles storage and memory management of objects - so that for instance you could use your memory manager on @drazzy's EEPROM. Hopefully when we've got a few memory drivers that work the same way, we can maybe pull your code into a module that will work with them? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by @allObjects Exactly: the ability to retrieve previously stored strings and the ability to temporarily store strings. The pubs I made is to give you some head start of the logic... I wanted to explore the techniques/logic in a higher lever environment. An intermediate layer is the management of the memory, and the bottom layer / foundation is the actual device driver. The basic functions are:
The functions in [square brackets] are just combinations of non-bracketed ones. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-14 by DrAzzy Wow, allObjects - that string data store is something that I'd need to study for a while to figure out - it pretty much went over my head on the first read. So - looking at the datasheet for that FRAM chip, it sounds like it's meant to be a drop-in replacement for the AT25. The same manufacturer also makes one that uses I2C and looks to a drop-in replacement for the AT24. (The FM24V02), and Fujitsu's MB85RC256V looks to use the same pinout and is probably the same way. Also, there ARE larger FRAM's in the same series - there are FM2xV05 (512), V10 (1mbit) and V20 (2mbit) available in SOIC-8 package, albeit at a truly eyewatering price. Try using my AT25 module to interface with your FRAM and see if it works (the writes will of course be slow, but if all that's needed is simplifying the write process and removing the delay, that makes converting the module to FRAM trivial. My module will NOT work on the V10 and V2o ones, but support for this in both AT25 and 24 is coming with the eeprom overhaul I mentioned. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-22 by DrAzzy For the new eeprom modules (requires recent v72 and the new eeprom modules)
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-22 by @allObjects
No wonder, you were at that time deep down in the details of the HW of EEPROMS. Coming back up into application (close) layers, it will be a breeze for you (and you may tell me where my doc needs to be fixed/enhanced, since I'm not of English mother tongue nor eloquent writer... JS is closer to me than English).
I'm fairing about the same way when looking on things coming out of your fine 'micro brewery' - on first reads. After taking the time diving into it, I get the exquisite taste. Thanks for your contributions!
Speed in accessing FRAM/MRAM depends on the needs of the application. To get/set config data and user data, the current speed is good enough, because the amount of data is not that critical - and there is most of the time little visibility to the user - if there is even one - to notice and complain about sluggishness. Speed is required only in graphics because of the amount of data and because visibility. My code to communicate with the FRAM was quite similar to your EEPROM code, even with the size limit because of (simplified) 2 bytes addressing with (singed) int. I can easily see speeding up the memory read/write process by using "compiled"; option, or moving the critical code into assembler all together... especially the part that shows patterns for what DMA was implemented: get a particular amount data at a particular address from one SPI (or other wise connected) device to another one (and back), like in my graphical app where screen buffer has to be saved for an overlay/pop-up - and then restored. For fast handling of those transfers, some chips have the hold pin, which keeps state, while the partial data just read is written to the other device... I was thinking along these lines to have some software DMA module implemented:
I can picture to use DMA on an intermediating Espruino in the data line between just any 'devices': sensors and internet... etc. I can also picture an option where the dma is setup to be trigger by a sourcing device to transfer the data to the sinking one... (...with the classical 'null' device... Piggy-backing existing (module) code with extending or overwriting is always an option, and if this is not good enough or too cumbersome, I could see different implementations for a particular layer in the existing software stack... assuming it is layered/a stack and not a monolith... ;-). On a micro - on the other hand - a few terse, dedicated modules go further than larger, elaborate multi-purpose components. Therefore, a good and easy way to compose all kinds of needed specific functions and just those form very smaller ones is key, and require("module") is the key enabler for that.
My FRAM is soldered onto my Espruino board. Will re-jumper-cable it and and do some testing, and also look at my x-code for the string memory. The string memory x-code needs transformation towards the available memory read/write interface with its data converters before I go and move it onto Espruino. I'd like to have also a test-bed/test for the sting memory store. And last but not least, implement an optional key function: store and read back by key rather than physical address (related object id)... even if it is just to get those physical addresses of the objects on init after a (re)boot. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-23 by DrAzzy The new AT24/25 modules will fix the slow writes problem on FRAM - just pass 0 as the page size. I just (now) fixed a problem with FRAM writes in the new modules that I introduced last night, they're up on my github now https://github.com/SpenceKonde/EspruinoDocs/tree/master/devices Don't use the old AT25 modules with your FRAM, write performance will suck :-P The idea of SPI DMA is interesting - I didn't realize that the STM32 chips supported it until I saw how the TV display code was done. I wonder how much it would buy us in terms of performance? Since I use a B&W LCD at 64x128 px, I don't have as much of a problem getting the data to it. ... I just had an idea (not necessarily a good one) - say you stored the entire blob of data to send to the display in the FRAM. Connect SO of FRAM to SI of display, with tristate buffer in the middle, then you could send the address, open the tristate buffer and assert screen's CS, and then read the command from EEPROM back straight into the display. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-23 by Manxome @allObjects, your English is excellent, hence your JS must be exquisite! |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-23 by @allObjects @drazzy, looked at the module and wondered why for each byte the address has to be written - have to say that I have not studied the EEPROM api. For FRAM resetting the address for each byte is not required - see](http://forum.espruino.com/comments/11937103/). Are you referring to that when talking about? Yep, the page size of 0 takes brilliantly care 0f skipping the wait cycles.
You just invented Super-Direct Memory Access! I did not think about that... for cases where the memory is equally or larger in size, this really works: preparing the next image/page in the memory and then update the display at once. For my case, the FRAM I have is a bit too small: 32,768 bytes FRAM vs. 172,800 byte display buffer (=230 x 320 pixel * 2bytes in ILI9341... and spending a lake of tears for 6 FRAMS and also giving up 6 CS lines, will become a sea of tears... nope. I need only a few KB to ship back and forth areas of the display. For that, I have to set two different addresses, and also connect SO of display to SI of FRAM. Super Direct DMA for save and restore... really cool idea. What I already thought of - but not had the time this morning to write about - was to have a DMA Hub... inspired by the 0 and 1 to indicate source and destination in my envisioned A_to_B API. Since the process would anyway not be multi-threaded, I could just add any communicating device to a hub, and then trigger the transfers between the from-tos with device#X to device#Y. For streaming devices, the address would be -1 (MSB = 1) or something other detectable (with 2 byte address and MSBit of MSByte a control bit, 32KB is the max addressable... just what my FRAM gives me. Note that not all devices have a single dimension address, even though they may support continuous read and write across dimensions, such as the ILI9341 display controler. This display controller requires to provide row and column address, but does auto increment/reset columns and rows.... |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-23 by @gfwilliams In terms of 'software DMA' - there are actually the 'pipes' that you can use for files, that you should be able to apply to SPI - at least for writing. Not that I've tried that though! It could end up being quite interesting. Hardware DMA would be cool - I'd been meaning to pull the DMA code from the TV stuff out into something more general purpose. I wonder how useful it is for stuff like LCDs though, as you still have the problem of not having enough RAM. I do have an interesting solution to that - but I'll stick another post up about it, |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-23 by DrAzzy @drazzy, looked at the module and wondered why for each byte the address has to be written - have to say that I have not studied the EEPROM api> It certainly shouldn't be writing the address for each byte, unless I screwed up... There are 2mbit FRAM chips ;-) Only $15-20 a pop and non-stock @ digikey. No sea of tears - just red ink. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-29 by d0773d I have a question which may have already been answered, but this thread is already long and way over my head. Could these chips be used instead of the microcontrollers internal memory to execute memory intensive code, to hopefully fix the out of memory issue while running a memory intensive script? |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-29 by DrAzzy Take a look at the writeup here, which has been updated to new version of the AT24 modules: http://www.espruino.com/run_code_from_eeprom You can store code on the EEPROM, and then load it off the EEPROM and execute it immediately - so you still need the memory free to hold the function while it's running, but not any other time. That's what this does. And you can of course write large blobs of data to the EEPROM, and then read it later when you need it - in this case, you're swapping stuff in and out of RAM. You can't have it access the EEPROM like it can main memory - you'll always have to swap stuff in and out. But these can help with it, by storing some function code on the EEPROM (in my projects, code is where much of my memory goes). It's probably significantly slower, making it best for code not frequently used. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-29 by d0773d Thank you for summing this subject up in layman's terms :-) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-01-12 by DrAzzy
Say you want to save memory storing functions (particularly verbose ones). I plan to do this for answering HTTP requests, since that's something that can involve a considerable amount of difficult-to-shrink code (the strings that need to be returned, mostly), yet is something that likely is not used all that often, so it can afford to drag a little bit.
Let's start by adding this function to our eeprom object:
Here, starting at 0x0400, we'll start storing a table of indices for functions stored on the eeprom, 4 bytes per (2 each for starting address, and length). We'll need some helper functions to load the EEPROM, and see what's on it. Setting all 4 bytes of the index entry for a function marks that function ID as unused, though we look at the third byte, because if that's 255, the result is unambiguously invalid.
And of course, the first function we add can be this one:
Length is like 1160 bytes.
Beta Was this translation helpful? Give feedback.
All reactions