Definition of a flat string and example

[espruino-discuss]

Posted at 2018-10-12 by Robin

Thr 2018.10.11

re: http://forum.espruino.com/conversations/316941/
'You can even access data stored in flat strings:'

What is a 'flat string' and how is it different? Definition, documentation please?
Is it just a contiguous block, null terminated? Is there a size limit?

Skimmed over

http://forum.espruino.com/conversations/316409/#comment14077573

It appears that 'flat string' may be specific, as a quick search only uncovered one reference, other than Espruino.com

https://dxr.mozilla.org/mozilla-central/source/js/src/jsapi.h#4035
'Flat strings and interned strings are always null-terminated,'

Having issues with var addrsrc = E.getAddressOf(src,true);
if(!addrsrc) throw new Error("Not a Flat String");

irratically returning 'Not a flat string', loading same data each run.

Working with

var buffer = new ArrayBuffer(8);
var n = new Uint8ClampedArray(buffer, 0, 4);

and E.compiledC

Is a flat string somehow related to Uint8ClampedArray
Not mentioned however in

https://www.espruino.com/Reference#t_Uint8ClampedArray

Is flatAddress just a T/F flag and not the address of the variable? This implies that param should always be true, e.g. why would you supply 'false' if you are attempting to get the address of (E.getAddressOf) the variable supplied? Seems redundant.

https://www.espruino.com/Reference#l_E_getAddressOf
flatAddress - If a flat String or flat ArrayBuffer is supplied, return the address of the data inside it - otherwise 0

I have a hunch that the size of the referenced address passed or returned is causing pointer (my code) offset errors.

[gfwilliams]

Posted at 2018-10-12 by @gfwilliams

I've made a note to update the Performance/Internals pages, but basically:

• In Espruino, ArrayBuffers actually store their data in a String behind the scenes
• Normal Strings are 'chained' - this is mentioned in the performance/internals pages, but basically it means a String can be fragmented all over memory. When you write "Hello" that's what you get, because Espruino didn't know the size of the string up front as it was parsing it as it went along.
• If Espruino knows the size of the string up front, there is enough contiguous free space, and the String is long enough that it is more efficient to use a Flat String* then Espruino will allocate a flat string - which is basically a single JsVar as a header, followed by N other JsVars that store the raw data.
• Flat Strings can take a while to allocate, and because of the fixed JsVar header they're not as efficient for small arrays. There's also no speed advantage even when accessing a small Flat String vs a small string - which is why they're allocated only when required.

So in your case, your arraybuffer just isn't big enough that it makes sense to allocate it as a flat string. Make it size 32 or something and it'll work.

You can however use E.toString to make Espruino create a flat string specifically for your data - for example E.toString(new Uint8Array(8)). You could also then use E.toArrayBuffer on that String to create an ArrayBuffer that actually references the Flat String itself.

[gfwilliams]

Posted at 2018-10-12 by @gfwilliams

Just to add, what I suggest (E.toString) is actually what's in the example right next to the 'you can even use flat strings' post that you linked right at the start of your post - so that would be a good starting point.

[espruino-discuss]

Posted at 2018-11-05 by asez73

Hi,
As a complementary question to those flat strings handling, I wonder if there is an efficient way to store a standard string (coming from usart for instance) to a pre-allocated flat string without using looping technics?.
This is in order to avoid the allocation time each time my pico receives some complete data packets.

More explicitely, the pico receives some data from usart.
These chuncks are concatenated in Serial.on('data',...) event handler.
So far, these are standard javascript strings.
Once there is enought data, it is sliced and can then be used in a flat string to create an ArrayBuffer resliced itself and finally accessed internally through a Dataview.

The thing is an Ublox M8U gps device using their binary protocol: very c structs friendly and only bytes/words/long words properly aligned. You can see this to get a clear idea. Here, the interest is that payloads are almost all of a fixed length (DMA in sight), very compact (less bytes on usart fro more data) and no fancy sexagesimally odd computations on latitudes/longitudes and so on!

My goal is to preallocate a few different dataviews of predefined length and then copy the incoming data from its string format to the dataview after all checksums and length controls of course.
The gps data will be accessed through getters using v.getUint32(buffer, true) and others as usual with dataviews.

A side approach would be to use inline C for setters and getters but what in Espruino (javascript)?

A dream would be to receive data from serial to an ArrayBuffer (kind of DMA) and only wake up when the serial buffer has enough bytes to analyse: so sleep walking while the interpreter is in deep sleep mode.

[espruino-discuss]

Posted at 2018-11-06 by Robin

Mon 2018.11.05

Hi @asez73,

'efficient way to store a standard string . . . . to a pre-allocated flat string without using looping technics?.'

While I hope I understand the gist of your request, after the 'light reading' of that struct ;-)

To also avoid looping, I had to rely on @gordon for the class hierarchy for this snippet, could this be what you are after?

array.set(...)

from class

function ArrayBufferView.set(arr, offset)

It appears from your code examples, you may be light years away from my skills, but I was able to get FlatStrings, inline.C working with plain JavaScript. Array buffers aligned with FlatStrings accessible both by inline.C and direct array access for comparison. A simple concise summary of other docs and what is at:

https://www.espruino.com/Reference#

> 'A side approach would be to use inline C for setters and getters but what in Espruino (javascript)?'

Yes, what you envision can be done that way. My method was required for fast <50msec (goal) access with large 1000byte strings, your
requirement might be closer to a sort of DMA

ref: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView

If I missed the concept using struct <--> DataView and concise memory allocation, others pipe in. . . .

Although I feel you are of the type that would tinker with code to seek your own solutions, should you desire some basic snippets for your perusal, I'll post them here.

Robin


EDIT - while searching the forum

http://forum.espruino.com/conversations/316959/

It appears you may already have the ammo you need from this example #1 of yours from nine months ago; Javascript usage of data stored-accessed in a Flat String

[gfwilliams]

Posted at 2018-11-06 by @gfwilliams

I think there are a few things that could help you out - some of which Robin already mentioned:

• You can do a = new Uint8Array(10);a.set("Hello") to write direct into an ArrayBuffer - it'll be pretty fast. I'm not sure accepting a string is spec compliant but it works great on Espruino.
• new DataView(uint8array.buffer) is very fast - no copies or anything
• E.toString will create a flat string from whatever you give it - but the allocation of a flat string can be slow
• E.toUint8Array/E.toArrayBuffer may also help
• it should be reasonably efficient to append to a string as long as the string isn't too big

But yeah, doing something like:

var buf = new Uint8Array(200);
var idx = 0;
SerialX.on('data',function(d) {
  buf.set(d,idx);
  idx += d.length;
  // ...
});

works fine

[espruino-discuss]

Posted at 2018-11-14 by asez73

HI @robin and @gordon,

Thanks for your help, it did worked and I globally saved some time AND events stack depth.
Now I don't have to separate in time, by emiting events, concatenation, slicing and storing from the Serial.on handler: it just happens fast enough without throwing any event to my Gps object.
In some cases, I could receive enough messages and emit one event per messages, which stopped the program with strange corrupted data in the line received. The result was a buffer way too long (32Kb) and other mysterious unvalid gps data.

Now, I am more on 2 colateral problems:
1-one in Javascript (Serial.pipe chunk parameter in order to limit the number of calls to handler of the input data and get at least any message header at once),
2-one in hardware (Pico powered by 3.7 V battery, deep sleeping as long as possible, using the B0 mosfet with the supplied 3.7 V)

So I think we are far enough of flat strings and should open 2 other threads to handle those points.

Thanks again!