WIP - FIxing ctpp_censustract_variables

[SPTKL]

• adding documentation to CTPP tables
• there are two potential methods here
  c451869...3479587
• c451869: slightly simpler, however say if one census tract doesn't have a certain mode (1-18), we will not have that row.
• 3479587: more verbose, however say if one census tract doesn't have a certain mode, this commit will populate it as value = 0, moe = 0 --> one to one replication of the python method, exact same number of rows in output

Also it seems like previous python based method has issues in calculating MOEs:

select 
	a.geoid, a.variable, 
	a.value, b.value as val_new,  
	a.moe, b.moe as moe_new
from ctpp_censustract_variables."2012_2016" a, 
	ctpp_censustract_variables."2020-07-22" b
where a.geoid||a.variable =  b.geoid||b.variable
and a.moe != b.moe

(moe) trans_auto_total 
	= sqrt(
		trans_auto_2 ** 2 +
		trans_auto_3 ** 2 + 
		trans_auto_4 ** 2 +
		trans_auto_5_or_6 ** 2 +
		trans_auto_7_or_more **2 +
                 trans_auto_solo ** 2 )
	= sqrt(126**2 + 50**2 + 
		16**2 + 13**2 +
		 0**2 + 422**2) 
	= 444

The previous calculation yields 11, which seems to be wrong

[mgraber]

Do we know why certain tracts have missing data? Typically in survey data 0 means people were surveyed and none responded "yes." There are 0 estimates in the raw data, which makes me think that if the missing records were 0, they would have been included. Is it possible to fill NULLS with 0 just for calculating the sum but not when outputting the final table?

[SPTKL]

The raw data doesn't have all 18 modes for all census tracts. we should probably ask labs for comments, because if we remove the 0 records, it might break the front end.

[mgraber]

How about:

SELECT
        a.geoid,
        CASE
        	WHEN a.value::numeric = 0 AND b.moe::numeric = 0
        		THEN NULL
        	ELSE a.value::numeric::integer 
        END as value,
        CASE
        	WHEN a.value::numeric = 0 AND b.moe::numeric = 0
        		THEN NULL
        	ELSE b.moe::numeric::integer 
        END as moe,
        a.variable
    FROM VAL a
    JOIN MOE b
    ON a.geoid = b.geoid 
    AND a.variable = b.variable