luajit: add decoder #709

dlatchx · 2023-06-20T16:43:46Z

a decoder for LuaJIT bytecode/precompiled files

wader

Thanks! Did some quick review and will do some more later. I think we should be able to add some custom decode function to replace some FieldAnyFn and maybe kgcType could use some standard mapper?

Overall looks good i think and thanks for including tests!

BTW you can add a luajit.md with author add embedd it, see wasm.go/wasm.md for example. Then it will be part of the documentation and also available in the CLI help.

wader · 2023-06-20T16:57:56Z

format/luajit/luajit.go

+		d.FieldValueBool("BE", flags&0x01 > 0)
+		d.FieldValueBool("STRIP", flags&0x02 > 0)
+		d.FieldValueBool("FFI", flags&0x04 > 0)
+		d.FieldValueBool("FR2", flags&0x08 > 0)


Field names are usually lowercase as it's the convention in jq

Can also use 0b... notation if you want to be explicit about bit positions

wader · 2023-06-20T17:05:04Z

format/luajit/luajit.go

+	ok := bytes.Equal([]byte(magic), []byte{0x1b, 0x4c, 0x4a})
+	if !ok {
+		d.Errorf("invalid magic number")
+	}


Think you can replace this with d.FieldStr("magic", 3, d.AssertStr("..")) or d.FieldRawLen("magic", 3*8, d.AssertBitBuf([]byte{...})) which will do the compare and also the magic will be shown as "valid"

wader · 2023-06-20T17:06:27Z

format/luajit/luajit.go

+
+	if !di.Strip {
+		namelen := d.FieldU8("namelen")
+		d.FieldStr("name", int(namelen), encoding.Nop)


Is encoding.Nop important? can skip if it's ok to be decoded as UTF-8

wader · 2023-06-20T17:10:06Z

format/luajit/opcodes.go

+func init() {
+	for i := 0; i < len(opcodes); i++ {
+		bcOpSyms[uint64(i)] = opcodes[i].Name
+	}


Wonder if you can make the BcDef implement scalar.UintMapper, i think the formatEntries type in msgpack.go does something similar. If so you can skip init and bcOpSyms

dlatchx · 2023-06-20T23:09:02Z

I think we should be able to add some custom decode function to replace some FieldAnyFn

I added more decode function, but is still need FieldAnyFn, because LuaJIT's format makes a weird use of ULEB128 encoding, and I want to map the bits of the two 32bit-max ULEB128 to the resulting 64bit Field. (see LuaJITDecodeI64(), for example)
Maybe there's another way to do that ?

dlatchx · 2023-06-20T23:09:41Z

... and maybe kgcType could use some standard mapper?

The problem is that we need a default case: 0 to 4 are types with know size, but 5 and more indicates a string and its size:
5 is a 0-length string, 6 is a 1-byte string, ect.

I don't think we can use a UintMapSymStr here.

wader · 2023-06-20T22:57:33Z

format/luajit/testdata/simple.fqtest

@@ -0,0 +1,274 @@
+$ fq dv -d luajit simple.luac


Could you add a comment to the .fqtest files or a README.md or Makefile how to generate the luac files?

wader · 2023-06-20T22:58:46Z

format/luajit/luajit.go

+		format.LuaJIT,
+		&decode.Format{
+			Description: "LuaJIT 2.0 bytecode dump",
+			DecodeFn:    LuaJITDecode,


The magic is quite unique so if you think it make senes you could add the format to the probe group

wader · 2023-06-20T23:01:12Z

format/luajit/luajit.go

+	interp.RegisterFormat(
+		format.LuaJIT,
+		&decode.Format{
+			Description: "LuaJIT 2.0 bytecode dump",


Possibly skip the "dump" part or are .luac files known as dump files?

wader · 2023-06-20T23:17:31Z

I added more decode function, but is still need FieldAnyFn, because LuaJIT's format makes a weird use of ULEB128 encoding, and I want to map the bits of the two 32bit-max ULEB128 to the resulting 64bit Field. (see LuaJITDecodeI64(), for example) Maybe there's another way to do that ?

I see, yeap that might be tricky.

The problem is that we need a default case: 0 to 4 are types with know size, but 5 and more indicates a string and its size:
5 is a 0-length string, 6 is a 1-byte string, ect.

I don't think we can use a UintMapSymStr here.

Aha that makes sense, i guess you could use a range mapper but maybe not worth it.

I added some more comments but getting a bit late here so will continue tomorrow. Think we're close to something mergable. Thanks for the patience!

wader · 2023-06-21T07:43:19Z

format/luajit/luajit.go

+			Description: "LuaJIT 2.0 bytecode",
+			Groups:      []*decode.Group{format.Probe},
+			DecodeFn:    LuaJITDecode,
+			Dependencies: []decode.Dependency{


Dependencies you can skip, it's if the format decoder itself want to decode something using some group

wader · 2023-06-21T07:47:54Z

format/luajit/luajit.go

+			})
+
+			if !di.Strip {
+				d.FieldArray("debug", func(d *decode.D) {


When i played around with the decoder i noticed that data in debug has some strings, is there any known structure to it or should be threaded as opaque blob? if blob should the array be replaced with d.FieldRawLen("debug", debuglen*8)?

wader · 2023-06-21T08:38:15Z

format/luajit/luajit.go

+	}
+}
+
+type kgcType struct{}


One variant is to do something like:

type fallbackUintMapSymStr struct { fallback string scalar.UintMapSymStr } func (m fallbackUintMapSymStr) MapUint(s scalar.Uint) (scalar.Uint, error) { s.Sym = m.fallback return m.UintMapSymStr.MapUint(s) }

and then

kgctype := d.FieldULEB128("type", fallbackUintMapSymStr{ fallback: "str", UintMapSymStr: scalar.UintMapSymStr{ 0: "child", 1: "tab", 2: "i64", 3: "u64", 4: "complex", }, })

and use it for ktabType also

Maybe the scalar package should have something like this if its common hmm

wader · 2023-06-21T08:45:45Z

format/luajit/luajit.go

+}
+
+func LuaJITDecodeI64(d *decode.D) {
+	d.FieldAnyFn("i64", func(d *decode.D) any {


For some other format decoders that do something similar i've decode the value as a "value" field, see msgpack or bson etc. Maybe that would make sense here also? for complex maybe the value field could an extra d.FieldValueAny("value", ...)?

wader · 2023-06-21T09:06:14Z

format/luajit/luajit.go

+	interp.RegisterFS(LuaJITFS)
+}
+
+// reinterpret an int as a float


Trying to understand how this works. So all numbers are floats but encoded i two parts where the hi part will be zero for < 32 bit integers? and hi might be > 2^32 so hi << 32 + lo will be > 2^64 and needs to be float? but looks like we only care about 8 bytes when reinterpreting? hope that made sense :)

wader · 2023-06-21T22:29:31Z

format/luajit/luajit.go

+		// make it a signed integer
+		lo_data_i32 := int32(lo_data_u32)
+
+		// return a larger type to make fq happy


🕺💃 maybe there should be smaller number types hmm, should probably do another round of profiling memory soon

wader · 2023-06-21T22:36:19Z

format/luajit/luajit.go

+		})
+
+		// TODO: find out more about how to decode these strings
+		d.FieldArray("anotations", func(d *decode.D) {


typo: annotations

👍 with more debug structure

wader · 2023-06-21T22:38:08Z

Lint not happy about unused probeGroup

Think i'm happy to merge if your happy too?

dlatchx · 2023-06-21T23:32:46Z

it's good for me

wader · 2023-06-21T23:45:53Z

Thanks!

luajit: initial support

1afdf8b

wader reviewed Jun 20, 2023

View reviewed changes

dlatchx added 8 commits June 20, 2023 23:02

go fmt

db2e621

luajit: lowercase flags

29ab66b

luajit: magic number: raw bits, check with assert

e44f5c0

luajit: check binary.Read() error

32300a3

luajit: use UTF8 strings

08ae661

luajit: opcodes: implement scalar.UintMapper

715f850

luajit: add luajit.md

bdf158b

luajit: split in smaller decode functions

52ce818

wader reviewed Jun 20, 2023

View reviewed changes

dlatchx added 3 commits June 21, 2023 01:25

luajit: *.fqtest: add comments for generating .luac from source

b24063b

luajit: clarify description

a83576a

luajit: add to probe group

93c9696

wader reviewed Jun 21, 2023

View reviewed changes

dlatchx added 8 commits June 21, 2023 18:47

luajit: remove unecessary dependency

c3a123a

luajit: fallbackUintMapSymStr

3561c08

luajit: standardize field names (key/value/type ect.)

441d246

luajit: tests: improve coverage

eb819dd

luajit: improve debuginfo decoding

64c11be

luajit: explain LuaJITDecodeKNum, fix negative in bug

751ee5a

luajit: tests: rename lua source file

dd594f4

luajit: fix regression: (u64 vs i64)

5d9a08c

wader reviewed Jun 21, 2023

View reviewed changes

dlatchx added 2 commits June 22, 2023 01:27

luajit: remove unused variable

64c92da

luajit: typo

c42fb9e

wader merged commit 5605ebe into wader:master Jun 21, 2023
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

luajit: add decoder #709

luajit: add decoder #709

dlatchx commented Jun 20, 2023

wader left a comment

wader Jun 20, 2023

wader Jun 20, 2023

wader Jun 20, 2023

wader Jun 20, 2023

wader Jun 20, 2023

dlatchx commented Jun 20, 2023 •

edited

dlatchx commented Jun 20, 2023

wader Jun 20, 2023

wader Jun 20, 2023

wader Jun 20, 2023

wader commented Jun 20, 2023

wader Jun 21, 2023 •

edited

wader Jun 21, 2023

wader Jun 21, 2023 •

edited

wader Jun 21, 2023

wader Jun 21, 2023

wader Jun 21, 2023

wader Jun 21, 2023

wader commented Jun 21, 2023

dlatchx commented Jun 21, 2023

wader commented Jun 21, 2023

luajit: add decoder #709

luajit: add decoder #709

Conversation

dlatchx commented Jun 20, 2023

wader left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlatchx commented Jun 20, 2023 • edited

dlatchx commented Jun 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wader commented Jun 20, 2023

wader Jun 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wader Jun 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wader commented Jun 21, 2023

dlatchx commented Jun 21, 2023

wader commented Jun 21, 2023

dlatchx commented Jun 20, 2023 •

edited

wader Jun 21, 2023 •

edited

wader Jun 21, 2023 •

edited