Extra columns in pandas dataframe not supported #758
Comments
|
It turns out that because the pandas DataFrame has extra columns - index, therefore I wasn't able to load the DataFrame into the table. After I select precisely the atot table column from the dataframe, then loading works successfully. Is this the expected behaviour? |
HuifangYeo
added a commit
to atoti/notebooks
that referenced
this issue
Feb 23, 2023
tibdex
changed the title
|
I can reproduce. In 0.7.2 this works: import pandas as pd
import atoti as tt
session = tt.Session()
table = session.read_pandas(
pd.DataFrame({"int": [1], "string": ["A"]}),
table_name="Default",
)
table.load_pandas(pd.DataFrame({"float": [1.5], "int": [2], "string": ["B"]}))but in 0.7.3 it fails with: When the extra column is at the end: - table.load_pandas(pd.DataFrame({"float": [1.5], "int": [2], "string": ["B"]}))
+ table.load_pandas(pd.DataFrame({"int": [2], "string": ["B"], "float": [1.5]}))it works in 0.7.3 too. This is tracked in an internal ticket. In your specific case, a fix is to do: for chunk in rt_chunks_df:
- t = chunk.reset_index()
payment_tbl.load_pandas(t)since |
printhellohetal
pushed a commit
to atoti/notebooks
that referenced
this issue
Feb 24, 2023
* Update table joins to conditional join * Update real-time risk on Bitnami kafka repo url, quantlib calendar for USA and Atoti's mapping to condition * Update customer 360 use case from Atoti mapping to condition * Update SA-CCR to switch join mapping to conditions * Update XVA to switch join mapping to conditions * Update join mapping to condition for CCF * Fixed tsfresh and protobuf conflict issue for collateral shortfall forecast notebook. Update mapping to condition * Update mapping to condition for collateral shortfall monitoring * Update mapping to condition for intraday liquidity * Update mappings to condition for airline industry use case * Update mappings to conditions for baseball notebook * Update mapping to conditions for ca-solar * Update mapping to conditions for digital marketing * remove mapping for drug efficacy * Update mapping to conditions for food processing notebook * Update mapping to conditions for F1 * update mapping to condition for election nb * Update mapping to condition for global covid nb * Update object detection to download data and model, mapping to conditions. * Update mappings to conditions for pokemon nb * Update mapping to conditions for pricing simulation nb * Update mapping from mapping to conditions for sales-commission nb * Update mapping to conditions for twitter nb * Update mapping to condition for conditional-function nb * Update mapping to conditions for curr-cov-weighted-avg nb * Update mapping to condition for curr-coversion nb * Update mappings to conditions for introductory-tutorial nb * Update mappings to conditions for rollup-hierarchies nb * Reorg pandas chart * Update `user.roles` to user_service_client.individual_roles instead * Formatting and update cell metadata for notebooks * Upgrade Atoti CE to v0.7.3 * Workaround for atoti/atoti#758 * Format notebook
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Steps to reproduce
The notebook was working previously on 0.7.2.
The above code threw out the below exception when trying to load the data with
payment_tbl.load_pandas(t):I have tried to read the data directly into a pandas dataframe instead of chunks and I have been able to load the data successfully without issue.
main.zip
Environment
Logs (if relevant)
server.log
The text was updated successfully, but these errors were encountered: