Contributions with bad data

[moonshoes87]

As @njarboe suggested, I'm making an issue to keep track of contributions with bad data. These will be mainly contributions on which make_magic_plots.py has failed due to bad/incomplete data. I will add to this list as I find more problems.

14019 -- no site column in the sites table, and generally very incomplete sites table.

[moonshoes87]

14290 -- no sample columns in samples table, no locations table.

[moonshoes87]

14575 -- not sure if this is technically "bad" data, but the sites table is missing many entries for location. This is one of those places where it would be very helpful if minimal info like location name were propagated throughout the rows where available. Doing so in PmagPy is possible but slow to do every time. It also looks like there are 7 locations but only 1 is actually linked to any sites, which seems a little suspicious.

[moonshoes87]

15417 -- this has only a contribution table.

[rminnett]

Thanks for doing this, Lori!

[njarboe]

16454 - contribution ID missing

[moonshoes87]

Contribution 11821 has values in measurements.specimen that are not present in the specimens table.

It also has at least one missing value in measurements.dec (specimen SLB05.4a, may be others).

[moonshoes87]

Contribution https://earthref.org/MagIC/11846: no 'sample' column in samples table.

[moonshoes87]

Contribution
https://earthref.org/MagIC/14127 has no 'site' column in sites table

[moonshoes87]

Contribution
https://earthref.org/MagIC/15417 is missing a locations table (or anything besides a contribution table).

[moonshoes87]

https://earthref.org/MagIC/15640 is missing locations table (or anything besides contribution)

[moonshoes87]

https://earthref.org/MagIC/16273 is missing locations table (but has samples --> measurements)

Also has two samples tables, two specimens tables, two measurements tables -- this one definitely needs some attention.

[moonshoes87]

contribution 16302 is missing at least one value in measurements.dir_dec

[moonshoes87]

15736 has no locations table: https://earthref.org/MagIC/15736

[moonshoes87]

16072 has no locations table: https://earthref.org/MagIC/16072

[moonshoes87]

16320 has negative values in sites.vgp_lat with a space between the '-' and the number, i.e.: − 69.1. Should this be fixed for download? It means that Python, at least, does not correctly translate this value into a float. I will put a fix in PmagPy, but seems like this could be annoying for others as well.

Edit: Actually, I had to do some weird manipulations because of the unicode characters in these strings. Not sure what is going on with this sites table, but the vgp_lat field contained values like \u2212 which have to be translated.

[moonshoes87]

16338 has many blank fields in measurements.method_codes.

[moonshoes87]

16410 downloads with an extra header row in the sites table:

>>>>>>>>>>
tab delimited	sites
site	location	lithologies	lat	lon	elevation	dir_tilt_correction	dir_dec	dir_inc	dir_alpha95	dir_r	dir_k	dir_n_samples	vgp_lat	vgp_lon
														
Site	Locality	Lithology	Latitude	Longitude	Elev (m)	Direction Tilt Correction	Dec	Inc	a95	R	K	N	VGP Lat	VGP Lon
DB0708	Haas Paleosol	Greyish brown mudstone	39.41202	-104.34196	1874.31	0.00000	335	42	39.9	2.81	10.62	3	64	137.4
DB0707	Haas Paleosol	Olive brown mudstone	39.41200	-104.34194	1873.14	0.00000	327.5	54.9	23.9	2.93	27.6	3	64	167.2

16416 https://earthref.org/MagIC/16416 has the same problem.

[moonshoes87]

https://earthref.org/MagIC/16418

No contribution id in contribution table.

[moonshoes87]

https://earthref.org/MagIC/16497

Several controlled vocabularies in the sites table have incorrect entries. method_codes, result_quality, and result_type are all filled with 1s.

[moonshoes87]

https://earthref.org/MagIC/16501 has no locations table.

[moonshoes87]

16416 https://earthref.org/MagIC/16416 has the descriptive row included as well as the actual headers.

[moonshoes87]

Something wrong in the naming hierarchy (can't propagate locations down to the measurement level):

13742, 16279, 16426, 15444, 16335, 15349, 15897, 16240, 16619, 16450, 16308, 16501, 16238, 16291, 16497, 16515, 13709, 14359, 16334, 15890, 16452, 16358, 11943, 14868, 12450, 16609, 16263, 16273, 16505, 11881, 15221, 14614, 16416, 11189, 16269, 14575, 16421, 11883, 16353, 11821, 16301, 15435, 16508, 16280, 16305, 12638, 16258, 16237, 16233, 14891, 16410, 15551, 11906, 13538, 16624, 16411, 16529, 16313, 15803, 15040, 14384, 16626, 15461, 15085, 11773, 11929, 11846, 13969, 16618, 16623

Locations table has many blanks in the 'location' column:

11189

Problem with naming hierarchy and missing column (treat_temp):

13727
14809
16457
16015

quick_hyst.py LP-HYS method code present, but required column(s) [treat_temp] missing

15283
15840
16458
16460

No tables found:

16277