txiki-utils!

[houd1ni]

I've made the ffi-based txiki-utils package! (for #921 initially)
https://codeberg.org/houd1ni/txiki-utils
gonna add there more stuff gradually as needed. Bun's approach of attaching the whole node's parts looks wrong here imho.

btw how do you look at further improving of module resolution to use node_modules at least?
if it's too much out of the scope, then do you see any ffi injection or any "wrapper" or another technique that could add that potentially without breaking into internals ? maybe some kind of import maps on-the-fly generation ? to keep things properly at their places.

[saghul]

Cool! Note that you can use tjs bundle rather than invoking esbuild by hand.

As for node_modules, I have no intention to add support for that. The runtime supports ES imports and import maps. A bundler can take care of the rest, I don't want to add that kind of complexity.

[houd1ni]

looks like there are no options that could be passed into esbuild via tjs bundle, and i use define: { 'tjs.env.NODE_ENV': '${process.env.NODE_ENV}' } to build for dev and dist (production). for everything else the configs are nearly identical though.

[saghul]

I see. In that case you might need to run it by hand.

If you have ideas on a way to pass parameters directly to esbuild I'd be interested!

[houd1ni]

--minify is already being passed through. maybe somehow copy all the options there ? ... oh wait - there is (undocumented?) extraArgs thing! ... just tested it. works! There's a PR to make it more straightforward: #923

[houd1ni]

hi again!
is it possible to free mem allocated in ffi module by attaching it somehow to the txiki/Quick-NG GC ? Is it planned or how ?

There are three ways as i see them:

1. alloc in js like new Uint8Array() and pass to ffi. available ✔️
2. alloc in the ffi module, free by calling a callback. eg. the newrly available "explicit res-ce management" using stuff. available ✔️
3. alloc in the ffi module, but somehow attach it to the gc. ideally smth like Node's napi_wrap. ❔

[saghul]

I think a way to do it would be to create a weakref to the object you got from ffi, and then when it goes way you'd get the callback and you can clean the array.

That should work today without any extra code I think.

[houd1ni]

the xxh3 hash (especially modern XXH3 version) is quite a useful non-crypto hash. could it be added some day in tjs:hashing or is it better to add it there in the utils separately ?

btw how do you estimate ffi-crossing lag ? I'm thinking isn't there any need to batch some things. I've managed to find some optimizations on top of libffi here, but honestly couldn't estimate properly. point me please. (gonna make benchmarks anyway tho)

[saghul]

Is there any other programming language that has it in its stdlib?

[houd1ni]

Is there any other programming language that has it in its stdlib?

if I got you right, no, at first glance it looks like it's pure C: https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h
also there's usage examples from inside of C.

other language implementations also.

[saghul]

Yeah I don't mean bindings, but rather have it work without any extra dependencies out of the box, like the hashlib python library for example.

[houd1ni]

there's a set of functions: https://txikijs.org/docs/api/tjs-ffi.Variable.read
but what if I really need the whole array (with known length) ? how can I make that "intermediate array"? without manual memcpy by read.u8 calls, more ffi calls or creating a large enough output buffer from js ofc. honestly, I digged a lot before asking 😄

[saghul]

I'll take a look. I guess what we need here is a way to return a SharedArrayBuffer...

[houd1ni]

thanks! also the SharedArrayBuffer spec is quite large, especially regarding web-specific security stuff that is probably not applicable in txiki. I'd (personally) be happy if it would land as a transport from C to JS first then, maybe, sometime, in a while 😄 , it would be complete.

(just to remind, 4sure you know this) the CFunction returns NativePointer interface object that could be casted to a number, but still cannot be fed to any of Uint8Array constructors. so close!

[saghul]

Nevermind, I misspoke. It's not SAB, it will be a Uint8Array with external memory.

[saghul]

Can you give this a try? #958