IDB data processing

In the Access DB

data\Processed\veg\IDB_Version2\IDB_version_2.mdb The final IdbSummary table (one row per condition)

MAKE_Qry_MP_BASE > Tbl_MP_Base >
MAKE_Qry_MP_BASE_TREESUMMARY > Tbl_MP_BASE_TREESUMMARY

The final TreeliveSummary table (multiple rows for each species-size class in each condition)

MAKE_TREELIVE_SUMMARY_STEP1 > Tbl_MP_TREELIVE_SUMMARY_SPECIESSIZECLS
MAKE_TREELIVE_SUMMARY_STEP2 > Tbl_MP_FINAL_TREELIVE_SUMMARY

Once in postgres, fix stand_age code

    UPDATE idb_summary
    SET stand_age = Null 
    WHERE stand_age >= 99;

    UPDATE idb_summary
    SET stand_age = (stand_age * 10) - 5
    WHERE stand_age <= 20;

    UPDATE idb_summary
    SET stand_age = 250
    WHERE stand_age = 21;

    UPDATE idb_summary
    SET stand_age = 350
    WHERE stand_age = 22;

create fixtures as tab separated files

export from access and/or

COPY idb_summary TO '/home/vagrant/test/idb_summary.tsv' WITH NULL AS '';

Potential check_integrity errors

* trees_conditionvariantlookup has 507 cond_id that are NOT in treelive_summary.
  [2057, 2061, 32788, 32789, '...']
* trees_fvsaggregate has 507 cond that are NOT in treelive_summary.
  [115101, 99301, '...']

The implication is that we are growing 507 more conditions than necessary (not in treelive == never matched). Tbl_MP_BASE_TREESUMMARY contains 507 records that have AvgOfTpa = 0, etc. These records don't exist in the idb tree_live database BUT they do exist in the gnn sppsz_attr_all table. Hmm.

Solution is to remove them from conditionvariantlookup and from the fvsbatch runs, If not worst that can happen is extra plots got run that will never get matched.

TODO: Confirm that this doesn't affect anything that Dave is doing. There shouldn't be any way to generate treelists from them unless Dave is using some other database or not using the lookups.

TODO: cross reference WC file names with lookup

* trees_conditionvariantlookup is missing 6532 cond_id that are found in treelive_summary.
   [3174, 3762, 36762, 36764, '...']

* trees_fvsaggregate is missing 6532 cond that are found in treelive_summary.
  [1, 3, 7, 8, '...']

Might not big deal... nearest neighbor match does a join with the lookup anyway so there is no chance of a condition showing up as a match without a corresponding fvs run. Still, it raises the issue of how the Tbl_MP_BASE_TREESUMMARY can be so far off the tree_live table. Might indicate that we didn't grow all the possible plots

* trees_fvsaggregate has X cond that are NOT in trees_conditionvariantlookup.
  [115101, 99301, '...']
* trees_fvsaggregate is missing X cond that are found in trees_conditionvariantlookup.
  [32768, 32769, 32770, 32771, '...']
* trees_fvsaggregate has 2 cond that are NOT in idb_summary.
  [115101, 99301, '...']
* trees_fvsaggregate is missing X cond that are found in idb_summary.
  [32768, 32769, 32770, 32771, '...']

Error We should always grow exactly the same conditions as defined by the condition lookup (idbsummary also being used to create the condition lookup).

Rx PN1 is not referenced in any variant's decision_tree_xml

The grow only rx is assumed to be __1 and grow only doesn't show up in the decision tree as it is created automatically by default.

Rx with type CI (Conventional, Even-aged, Short rotation) doesn't exist in Inland California and 
Southern Cascades (CA) but should according to RX_TYPE_CHOICES

RX_TYPE_CHOICES defines the typed rxs created by default. This one will just be ignored when myrxs are automatically created.