something wrong with the message type

[cchayden]

Has anyone else seen the following problem? It occurs intermittently,

I receive and parse a message. I get the message id from message['id']
I test:
if message_id in (1, 2, 3, 18, 19, 27):
do something

On the if statement, python throws an exception: list assignment index out of range (IndexError)

It is accompanied by an error: RuntimeWarning: tp_compare didn't return -1 or -2 for exception

This is confusing because the if statement is not doing an index assignment.
My guess is that internally python is comparing message_id with the values in the tuple, and one of them does not compare properly. Perhaps because message_id came from C++ and was not properly converted to a legal python value. When I try to print it, message_id prints as "6".

I started seeing this on Nov 13.
On Nov 15, there were six such errors around 00:42 UTC and another six at 12:04.

If it would help, I can get the raw messages that triggered these errors.

[schwehr]

Raw messages triggering trouble would definitely help.

[pwoods25443]

I have the same problem when testing message['id'] and I believe that I have isolated the bad message. In my case the bad message was parsing just fine and the error occurred on the following message. I have isolated it down to this, working from master:

import ais

# works
print 1 in (1,2,3)

# works
print ais.decode("63aIjB@0UBf`04@0@0000:00000000000000000", 2)

# fails after the print
print 1 in (1,2,3)

Output

True
{u'cap_reserved': [0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L], u'retransmit': True, u'seq': 0L, u'dac': 1L, u'mmsi': 244740681L, u'repeat_indicator': 0L, u'capabilities': [0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L], u'mmsi_dest': 2444010L, u'spare': 0L, u'ack_dac': 1L, u'fi': 4L, u'spare2': 0L, u'id': 6L}
True
Exception IndexError: 'list assignment index out of range' in <module 'threading' from '/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.pyc'> ignored

It appears that something happens in decode() that breaks python

[schwehr]

I'm going to write some tests for 6:1:4 messages.

// 6 1 4 205323000 !AIVDM,1,1,,A,633krv00OEGl04@0Hb00020000000000000000000000000000000000000,2*30
// 6 1 4 205317690 !AIVDM,1,1,,A,633kV>P0OEH@04@0@2000200000000000000000000000000T7L08611020,2*54
// 6 1 4 244850179 !AIVDM,1,1,,B,63aPN0h0UBfj04@0@2000200000000000000000000000000T7L08611020,2*09
// 6 1 4 235095657 !AIVDM,1,1,,B,63P=6J@0WGQt04@0Hb00020000000000000000000000000000000000000,2*1A
// 6 1 4 244110350 !AIVDM,1,1,,A,63`kD3`0UBf`04@0Hb00020000000000000000000000000000000000000,2*70
// '63aIjB@0UBf`04@0@0000:00000000000000000', 2

Note the non-zero data in some of the spare blocks :(

[jamtho]

In function

libais/src/libais/ais_py.cpp

Lines 409 to 436 in 538908f

	AIS_STATUS
	ais6_1_4_append_pydict(const char *nmea_payload, PyObject *dict,
	const size_t pad) {
	assert(nmea_payload);
	assert(dict);
	assert(pad < 6);
	Ais6_1_4 msg(nmea_payload, pad);
	if (msg.had_error()) {
	return msg.get_error();
	}
	DictSafeSetItem(dict, "ack_dac", msg.ack_dac);
	PyObject *cap_list = PyList_New(26);
	PyObject *res_list = PyList_New(26);
	for (size_t cap_num = 0; cap_num < 128/2; cap_num++) {
	// TODO(schwehr): memory leak?
	PyObject *cap = PyLong_FromLong(long(msg.capabilities[cap_num])); // NOLINT
	PyList_SetItem(cap_list, cap_num, cap);
	PyObject *res = PyLong_FromLong(long(msg.cap_reserved[cap_num])); // NOLINT
	PyList_SetItem(res_list, cap_num, res);
	}
	DictSafeSetItem(dict, "capabilities", cap_list);
	DictSafeSetItem(dict, "cap_reserved", res_list);
	DictSafeSetItem(dict, "spare2", msg.spare2);
	return AIS_OK;
	}

Two pylists of length 26 are constructed. However the for loop runs to index 128/2, and calls PyList_SetItem() with this index for both lists. This presumably writes beyond the end of a malloc'd block.

Changing the calls to PyList_New(128/2) removes the IndexError on my system.

[schwehr]

Thanks @jamtho

There is still lots more wrong with 6:1:4 starting with needing tests, but this should help.

[schwehr]

f2419b4 is a better 6:1:4, but probably could still be better... e.g. handle if there is a non-standard amount of spare.

[schwehr]

If anyone has test messages that are crashing libais, please create a new issue with them.