v1.3.0: Dynamic Columns
v1.3.0: Dynamic Columns
Improvements
-
Syncing now handles dynamic columns.
Syncing a pipe with new columns will trigger anALTER TABLE
query to append the columns to your table:import meerschaum as mrsm pipe = mrsm.Pipe('foo', 'bar', instance='sql:memory') pipe.sync([{'a': 1}]) print(pipe.get_data()) # a # 0 1 pipe.sync([{'b': 1}]) print(pipe.get_data()) # a b # 0 1 <NA> # 1 <NA> 1
If you've specified index columns, you can use this feature to fill in
NULL
values in your table:import meerschaum as mrsm pipe = mrsm.Pipe( 'foo', 'bar', columns = {'id': 'id_col'}, instance = 'sql:memory', ) pipe.sync([{'id_col': 1, 'a': 10.0}]) pipe.sync([{'id_col': 1, 'b': 20.0}]) print(pipe.get_data()) # id_col a b # 0 1 10.0 20.0
-
Add as many indices as you like.
In addition to the special index column labelsdatetime
,id
, andvalue
, the values of all keys within thePipe.columns
dictionary will be treated as indices when creating and updating tables:import meerschaum as mrsm indices = {'micro': 'station', 'macro': 'country'} pipe = mrsm.Pipe('demo', 'weather', columns=indices, instance='sql:memory') docs = [{'station': 1, 'country': 'USA', 'temp_f': 80.6}] pipe.sync(docs) docs = [{'station': 1, 'country': 'USA', 'temp_c': 27.0}] pipe.sync(docs) print(pipe.get_data()) # station country temp_f temp_c # 0 1 USA 80.6 27.0
-
Added a default 60-second timeout for pipe attributes.
All parameter properties (e.g.Pipe.columns
,Pipe.target
,Pipe.dtypes
, etc.) will sync with the instance every 60 seconds. The in-memory attributes will be patched on top of the database values, so your unsaved state won't be lost (persist your state withPipe.edit()
). You can change the timeout duration withmrsm edit config pipes
under the keysattributes:local_cache_timeout_seconds
. To disable this caching behavior, set the value tonull
. -
Added custom indices and Pandas data types to the Web UI.