Speed up codelist tests

[andylolz]

Codelist tests are among the slowest dashboard tests, so it’s worth investigating how to speed these up.

This PR makes two changes, and results in significant efficiencies (benchmarking suggests this could make python calculate_stats.py loop about 20% faster):

1. Codelist tests are currently done at element level. This means lots and lots of slow xpath calls. It’s way more efficient to perform these tests at file level.

2. At present, codelist values are computed and then recomputed:

   IATI-Stats/stats/dashboard.py

   Lines 539 to 553 in 278c0c4

   	@returns_numberdictdict
   	def codelist_values(self):
   	out = defaultdict(lambda: defaultdict(int))
   	for path in codelist_mappings[self._major_version()]:
   	for value in self.element.xpath(path):
   	out[path][value] += 1
   	return out
   	@returns_numberdictdict
   	def codelist_values_by_major_version(self):
   	out = defaultdict(lambda: defaultdict(int))
   	for path in codelist_mappings[self._major_version()]:
   	for value in self.element.xpath(path):
   	out[path][value] += 1
   	return { self._major_version(): out }

   If codelist tests are done at file level, we can cache the results to avoid recomputing.

[coveralls]

Coverage increased (+0.03%) to 40.567% when pulling 2263a35 on andylolz:element-to-file-rejig into ecc2398 on IATI:master.

[andylolz]

I’ve brought this PR up-to-date, since it sounds like the tech team is currently working on dashboard efficiencies, so this may be relevant.

I’ve just tested this, and the output is the same, but python calculate_stats.py loop is about 15% faster on my machine. It’s possible this uses more memory, so it might be worth profiling that.

[andylolz]

Doesn’t look like this will be merged; closing.