datapkg_to_sqlite fails to load all of EPA CEMS

[zaneselvans]

After doing a full ETL of all years and states in CEMS, the datapkg_to_sqlite script doesn't seem to load all of that data into the SQLite database. Rather, it only loads the last year of data into the database. However, the process terminates quickly, so it's probably not even attempting to load all the data. Suspect it's an issue with the iteration and/or partitioning...

[cmgosnell]

Hmmm I tried to recreate this issue with a grouped table and am getting a similar result. For me only the last resource described in the datapackage.json file is showing up in the sqlite database. @roll, is it possible that for the grouped table sqlite export is not appending the parts of the grouped table but just clobbering each of the earlier groups.

I generated data packages for CEMS with these parameters:

epacems_years: [2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014]
epacems_states: [ID,MT,CO,TX,IL]

It is being partitioned by state and year so each tabluar resource for the
hourly_emissions_epacems table is a part of the same group. Here is a description of one of the resources:

       {
            "profile": "tabular-data-resource",
            "name": "hourly_emissions_epacems_2005_id",
            "path": "data/hourly_emissions_epacems_2005_id.csv.gz",
            "title": "hourly_emissions_epacems",
            "encoding": "utf-8",
            "mediatype": "text/csv",
            "format": "csv",
            "dialect": {
                "delimiter": ",",
                "header": true,
                "quoteChar": "\"",
                "doubleQuote": true,
                "lineTerminator": "\r\n",
                "skipInitialSpace": true,
                "caseSensitiveHeader": false
            },
            "schema": {
                "fields": [
                    {
                        "name": "plant_id_eia",
                        "type": "integer",
                        "description": "The unique six-digit facility identification number, also called an ORISPL, assigned by the Energy Information Administration.",
                        "format": "default"
                    },
## a bunch of deleted field descriptions
                            ]
                        }
                    }
                }
            ],
            "bytes": 229820,
            "hash": "sha256:12d2998a68c0689c19061e001910519c7a19dfe6be3fbc8444f914d9a9ff4c9f",
            "created": "2019-10-22T19:58:40Z",
            "start_date": "2005-01-01",
            "end_date": "2014-21-31",
            "group": "hourly_emissions_epacems"
        },

[roll]

@cmgosnell
Could you please share something I can run to debug this issue?

PS.
Please make sure you're on the latest versions of tableschema and datapackage

[roll]

@cmgosnell
Has updating the libs versions helped?

[zaneselvans]

No, still getting the error with the most recent versions of the datapackage libraries. A very simple version with a couple of resources in a group seems to work as expected, but the simplest PUDL output that tests the behavior doesn't work. I'm trying to simplify that resource group output one step at a time until I get to a minimal example to share with you.

[cmgosnell]

Hey @roll! this was totally just our mistake, so don't worry about it. We are making a bunch of data source specific data packages and then squishing them together into one package, I set up a process for determining how to generate a new data package without duplicating elements of the metadata.... but I messed up and it was only grabbing one of the CEMS resources. It was a very simple fix once we figured out what was happening.