JSON array encoding fails on array of objects

[lucasdietrich]

Hi,

When I try to encode an array of objects. If the descriptors of the inner object doesn't cover all the fields of the object structure, then the array is wrongly encoded.

Normal case:

#define MY_ARRAY_SIZE 4u

struct mystruct_obj
{
	uint32_t a;
	/* Uncomment following line to get an incorrect result */
	// uint32_t b;
	uint32_t n;
};

static const struct json_obj_descr descr_mystruct_obj[] = {
	JSON_OBJ_DESCR_PRIM(struct mystruct_obj, a, JSON_TOK_NUMBER),
	JSON_OBJ_DESCR_PRIM(struct mystruct_obj, n, JSON_TOK_NUMBER),
};

struct mystruct_arr
{
	struct mystruct_obj items[MY_ARRAY_SIZE];
	size_t count;
};

static const struct json_obj_descr descr_mystruct_arr[] = {
	JSON_OBJ_DESCR_OBJ_ARRAY(
		struct mystruct_arr,
		items,
		MY_ARRAY_SIZE,
		count,
		descr_mystruct_obj,
		ARRAY_SIZE(descr_mystruct_obj)
	)
};

int encode_to(char *buf, size_t len)
{
	struct mystruct_arr arr = { .count = 0u };

	for (uint8_t i = 0u; i < MY_ARRAY_SIZE; i++) {
		arr.items[i].a = 3;
                // Uncomment following line to highlight that the encoder reads at incorrect offsets.
		// arr.items[i].b = 0xFFFFFFFF;
		arr.items[i].n = i;
		arr.count++;
	}

	return json_arr_encode_buf(descr_mystruct_arr, &arr, buf, len);
}

With the previous code, the json is properly encoded :

[
	{
		"a": 3,
		"n": 0
	},
	{
		"a": 3,
		"n": 1
	},
	{
		"a": 3,
		"n": 2
	},
	{
		"a": 3,
		"n": 3
	}
]

But if I uncomment the line uint32_t b;.
Here, "b" member is still not encoded in the JSON, but I would have expect the result to be the same, however I get the following result. Array is incorrectly encoded.

[
	{
		"a": 3,
		"n": 0
	},
	{
		"a": 0,
		"n": 0
	},
	{
		"a": 0,
		"n": 3
	},
	{
		"a": 3,
		"n": 2
	}
]

If I set "b" to "0xFFFFFFFF" in initialization loop, it seems that the value of "b" is used sporadically. This highlights that the encoder reads the values at incorrect offsets while encoding the array.

[
	{
		"a": 3,
		"n": 0
	},
	{
		"a": 0,
		"n": -1
	},
	{
		"a": -1,
		"n": 3
	},
	{
		"a": 3,
		"n": 2
	}
]

Is this just an undocummented unhandled case ? I do understand this case has not been foreseen.

Impact
Impact is that we cannot use C structure (for JSON encoding) which are not exhaustively described using descriptors. In my case, I will not be able to use existing structure if there are fields I don't want to encode, I would need to create structures dedicated to JSON encoding.

I haven't done much debugging yet, maybe I will dig into this when I have time.

Environment:

• Linux
• Zephyr v3.1.0
• QEMU x86

[mrfuchs]

Without having it debugged, I would say the reason is that arr_encode() iterates over the array items by incrementing the data pointer by elem_size.

zephyr/lib/os/json.c

Line 883 in 87d7c8f

field = (char *)field + elem_size;

elem_size is retrieved via get_elem_size() which reconstructs an element's size by the information given in the JSON descriptor. As b is part of the structure but missing from the descriptor, encoding will fail.

zephyr/lib/os/json.c

Lines 834 to 838 in 87d7c8f

	static int arr_encode(const struct json_obj_descr *elem_descr,
	const void *field, const void *val,
	json_append_bytes_t append_bytes, void *data)
	{
	ptrdiff_t elem_size = get_elem_size(elem_descr);

zephyr/lib/os/json.c

Lines 508 to 523 in 87d7c8f

	static ptrdiff_t get_elem_size(const struct json_obj_descr *descr)
	{
	switch (descr->type) {
	case JSON_TOK_NUMBER:
	return sizeof(int32_t);
	case JSON_TOK_OPAQUE:
	case JSON_TOK_FLOAT:
	case JSON_TOK_OBJ_ARRAY:
	return sizeof(struct json_obj_token);
	case JSON_TOK_STRING:
	return sizeof(char *);
	case JSON_TOK_TRUE:
	case JSON_TOK_FALSE:
	return sizeof(bool);
	case JSON_TOK_ARRAY_START:
	return descr->array.n_elements * get_elem_size(descr->array.element_descr);

[lucasdietrich]

Hi Markus,

Thank you for your quick analysis. After debugging this, your explanation perfectly matches my observations.

Now the question is: does this behavior deserves to be enhanced or not ? This is probably not a must have. But anyone could unintentionally end up with JSON descriptors not matching C structure. So telling explicitly somewhere in the documentation that we can't do that would be a plus.

I'll try to work on it (because I need it anyway), let's see if I can enchance this behavior without breaking everything.
I have a first dirty but working solution https://github.com/lucasdietrich/zephyr/commits/json_uncovered_struct.

---

Totally aside of this:
I'm actually a bit confused about what kind of value the "val" argument of arr_parse() and decode_value() should hold.
When debugging test case test_json_decoding_array_array of tests/lib/json, obj_parse() is called twice (recursively):

• When parsing the outer-most array with "val=0x10e000 <obj_array_array_ts>"

• When parsing the nested array with "val=NULL"

• "val" argument of decode_value() seems to be used only for array decoding.

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

[teburd]

Not stale, pending PR

[github-actions]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.