Question data column name can collide with reserved names in JSON representation of submission

[bufke]

Description

It's possible to collide question data column name with values in the submission data API

Steps to Reproduce

1. Create a form with question called _attachments (probably other colliding values work too)
2. Submit form

Expected behavior

Either _attachment is accepted and works or is rejected/changed

Actual behavior

The answer to the _attachment question does not show in the submissions data api at all. And this breaks the frontend when viewing details of the submission.

Possible solutions

• Having questions appear as root level values in the submission api is the root cause here. The best fix would be to restructure this API in a future v3. It's difficult to work with the data API without more structure. For example, it's hard to fetch all question answers from it.
• Attempt to detect and reject/change data column names with invalid (colliding) values. This might prove to be tedious and limited but is much easier to implement quickly.

[jnm]

Thanks. It's not actually an API problem but rather, depending on how you look at it:

• A problem the way kobocat parses the XML submissions into JSON. You can also add questions to your forms with other leading-underscore names like _submitted_by, and these will be mercilessly overwritten by kobocat in the JSON representation of the submission. The original response, however, will remain in the XML.
• Or, a failure to prevent people from creating forms using these reserved names.

_attachments and friends are written to MongoDB by to_dict_for_mongo():

kobocat/onadata/apps/viewer/models/parsed_instance.py

Lines 263 to 285 in 8b81f3e

	def to_dict_for_mongo(self):
	d = self.to_dict()
	data = {
	UUID: self.instance.uuid,
	ID: self.instance.id,
	self.USERFORM_ID: '%s_%s' % (
	self.instance.xform.user.username,
	self.instance.xform.id_string),
	ATTACHMENTS: _get_attachments_from_instance(self.instance),
	self.STATUS: self.instance.status,
	GEOLOCATION: [self.lat, self.lng],
	SUBMISSION_TIME: self.instance.date_created.strftime(
	MONGO_STRFTIME),
	TAGS: list(self.instance.tags.names()),
	NOTES: self.get_notes(),
	VALIDATION_STATUS: self.instance.get_validation_status(),
	SUBMITTED_BY: self.instance.user.username
	if self.instance.user else None
	}
	d.update(data)
	return MongoHelper.to_safe_dict(d)

It's been this way for over a decade, so as plainly unfortunate as it is, it's clearly not a major stumbling block for people using the tool.

Attempt to detect and reject/change data column names with invalid (colliding) values. This might prove to be tedious and limited but is much easier to implement quickly.

I think this could be done in the near term. The list of these reserved names is well defined.