Split TableTransformer.transform() into 3 phases #1900
Conversation
Within each dataset, there are often a set of standard operations that have to happen at the beginning and the end of each table's transformation. To allow those operations to be defined just once per dataset, I create 3 separate phases, defined as abstract methods in the AbstractTableTransformer class: * start_transform() * main_transform() * finish_transform() These are called by the (no longer abstract) TableTransformer.transform() method and should be defined by the dataset specific abstract table transformer. Of course they can also be overridden by the individual concrete table transformer when necessary. This commit also adds a decorator @cache_df() that enables the optional caching of the outputs of the individual in the self._cached_dfs dictionary for forensic / debugging purposes. This caching is only done if self.cache_dfs = True. By default it is False. The transform() method removes all cached dataframes after calling finish_transform() method unless TableTransformer.clear_cached_dfs is False. It is True by default.
zaneselvans
added
ferc1
|
@cmgosnell if I do any more work on the row dropper methods before you get back to your computer I'll add them to this PR, but I will try and make sure that each commit is a self-contained piece of functionality so you can look at them individually. |
Create a fuel_ferc1 table specific drop_invalid_rows method, which both makes use of the parameterized method inherited from the AbstractTableTransformer, and adds a separate method for identifying rows which we believe to be plant totals, rather than rows that pertain to individual fuels. This results in dropping more than 2/3 of all the records in the fuel_ferc1 table. I also made some name changes, trying to be a little more standard and (hopefully) informative: * remove_invalid_rows => drop_invalid_rows, since there were already several methods defined in the FERC 1 table transformers that do similar things in more specific circumstances, and they were named drop_*_cols or drop_*_rows. * cols_to_check => required_valid_cols and cols_to_not_check => allowed_invalid_cols to provide some indication as to what is being checked for (validity / invalidity) in relation to the other transform parameter (invalid_values) * DropInvalidRows => InvalidRows to follow the convention of the other TransformParams which are nouns describing their contents, while the methods & functions that they parameterize are verbs/actions. * Moved the documentation of the InvalidRows parameters into the class definition.
|
I've responded to your questions/comments and added better documentation of the Sorry about confusingly merging in some code changes from the I still think I prefer the |
There was a problem hiding this comment.
Thanks for the decorator docs. some of that seemed inferrable but its nice to make it clear.
On the naming.. I've tended to use "reverse notation" for many reasons. while its slightly less colloquial, it clearly groups elements and it is easier to use with tab completion.
…rows Implement drop_invalid_rows() for fuel_ferc1 table
Within each dataset, there are often a set of standard operations that have to happen at
the beginning and the end of each table's transformation. To allow those operations to
be defined just once per dataset, I create 3 separate phases, defined as abstract
methods in the AbstractTableTransformer class:
These are called by the (no longer abstract) TableTransformer.transform() method and
should be defined by the dataset specific abstract table transformer. Of course they can
also be overridden by the individual concrete table transformer when necessary.
This commit also adds a decorator @cache_df() that enables the optional caching of the
outputs of the individual in the self._cached_dfs dictionary for forensic / debugging
purposes.
This caching is only done if self.cache_dfs = True. By default it is False.
The transform() method removes all cached dataframes after calling finish_transform()
method unless TableTransformer.clear_cached_dfs is False. It is True by default.
For more information on how decorators work, I found this post useful