Validate all headers/fields when bulk import objects

[julianstolp]

NetBox version

v3.4.3

Python version

3.8

Steps to Reproduce

1. Open Prefix Bulk Import
2. Specify all required field options and add some dummy data
   prefix,status,testa,testb
10.10.10.0/24,active,a,b
3. Press Import
4. Import is done without any complains

Expected Behavior

Fields which are not stated in Field Options should throw an error, or a warning that some data was ignored.

Observed Behavior

Import is done without any complains, Looks like all data was imported correct, although some fields were ignored

[kkthxbye-code]

It this a regression? Did it used to throw an error? Are we sure this is actually wanted behavior before I look into this?

[julianstolp]

I just tried it in v3.09. It raises a Unexpected column header "testa" found. error.

[julianstolp]

Also checked versions 3.2.0, 3.3.0 and 3.4.0. The import works with version 3.4.0, before raises an Unexpected column header "testa" found. error.

[kkthxbye-code]

@arthanson - Can you take a look at this as it's part of the bulk import stuff you changed.

I tried just adding the following validation on L147 here:

netbox/netbox/utilities/forms/forms.py

Lines 134 to 153 in 56c7a23

	class CSVModelForm(forms.ModelForm):
	"""
	ModelForm used for the import of objects in CSV format.
	"""
	def __init__(self, *args, headers=None, fields=None, **kwargs):
	headers = headers or {}
	fields = fields or []
	super().__init__(*args, **kwargs)
	# Modify the model form to accommodate any customized to_field_name properties
	for field, to_field in headers.items():
	if to_field is not None:
	self.fields[field].to_field_name = to_field
	# Omit any fields not specified (e.g. because the form is being used to
	# updated rather than create objects)
	if fields:
	for field in list(self.fields.keys()):
	if field not in fields:
	del self.fields[field]

for header in headers.keys():
    if header not in self.fields:
        raise forms.ValidationError(f'Unexpected column header "{header}" found.')

But the ValidationError is caught here and discarded:

netbox/netbox/netbox/views/generic/bulk_views.py

Lines 461 to 462 in 56c7a23

	except (AbortTransaction, ValidationError):
	clear_webhooks.send(sender=self)

[jeremystretch]

We should consider JSON/YAML imports as well as CSV. If we're concerned about checking for extraneous CSV headers (and IMO we should be), then this functionality should exist for JSON- and YAML-formatted data as well. Unlike CSV, which specifies a single row of headers, the fields for each object expressed in JSON or YAML can vary with each object, so each object will need to be validated individually. I think it's reasonable to replicate this pattern for CSV data, even if it's not technically necessary.