Skip to content

Commit

Permalink
Adds in memory data saver
Browse files Browse the repository at this point in the history
So that one can do the following:

```python

to.memory(
     id="output_df",
     dependencies=outputs,
     combine=base.PandasDataFrameResult()
    ),
```
i.e. reuse any result builder and get that result return
in memory. This is in effect helps one to dynamically add/adjust
a result builder -- which I think is useful for development.
  • Loading branch information
skrawcz committed Oct 4, 2023
1 parent 619e63c commit 54f0639
Showing 1 changed file with 20 additions and 0 deletions.
20 changes: 20 additions & 0 deletions hamilton/io/default_data_loaders.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,10 +107,30 @@ def name(cls) -> str:
return "literal"


@dataclasses.dataclass
class InMemoryResult(DataSaver):
"""Class specifically to returning an in memory result without saving it anywhere.
Use this to get the result of a combiner easily; depends on their use.
"""

@classmethod
def applicable_types(cls) -> Collection[Type]:
return [Any]

def save_data(self, data: Any) -> Any:
return data

@classmethod
def name(cls) -> str:
return "memory"


DATA_ADAPTERS = [
JSONDataAdapter,
LiteralValueDataLoader,
RawFileDataLoader,
PickleLoader,
EnvVarDataLoader,
InMemoryResult,
]

0 comments on commit 54f0639

Please sign in to comment.