Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve Trades data handling #9065

Merged
merged 23 commits into from
Aug 18, 2023
Merged

Improve Trades data handling #9065

merged 23 commits into from
Aug 18, 2023

Conversation

xmatthias
Copy link
Member

@xmatthias xmatthias commented Aug 18, 2023 •
edited

Summary

Trades data should be handled as dataframes. That'll speed up most operations quite some, and avoid double-conversation on all formats except json, which load / store data as dataframes already.

Quick changelog

  • Add parquet dataformat support for trades data
  • Convert trades data handling to use dataframes

@xmatthias
Update Datahandlers to work with trades data as dataframes
4688240
@xmatthias
Update converters for trades dataframe handling
ac80a69
@xmatthias
Default should return a dataframe, not an empty list.
7ac9d33
@xmatthias
Update test for new trades handling
a595074
@xmatthias
Update further tests
02ee7f8
@xmatthias
don't cascade calls, that creates an additional call
53db254
@xmatthias
Keep existing trades_remove_duplicates for now
0be2250
@xmatthias
Fix pandas duplication detection, improve test
5d5cc71
@xmatthias
Fix trades duplicates if trade id is different
780f238
@xmatthias
trades_append should use dataframe, not lists
6fc1ee9
@xmatthias
Datahandlers should store data from dataframes
357b042
@xmatthias
Extract trades DF creation to converter function
d97d0e4
@xmatthias
Keep original timestamp in dataframe
26c89d8
@xmatthias
Ensure only relevant columns are stored
f69a776
@xmatthias
Set explicit dtypes
0fab65d
@xmatthias
Extract trades df type conversion
7577613
@xmatthias
Update download-trades method to work with dataframes
fa5f7d2
@xmatthias
Fix imports
fae742d
@xmatthias
Clean up no longer used method
5de3b9d
@xmatthias
Simplify trades_data storing
3eb3596
@xmatthias
Fix feather handler storing
8caadc4
@xmatthias
Add support for trades data in Parquet format
30064b4
@xmatthias
Improve and parametrize trades_data tests
3d8dcd1
@coveralls
Copy link

Coverage Status

coverage: 94.651%. remained the same when pulling 3d8dcd1 on trades_data_handling into d4bc736 on develop.

@xmatthias xmatthias added the Enhancement Enhancements to the bot. Get lower priority than bugs by default. label Aug 18, 2023
@xmatthias xmatthias merged commit aa1dcd1 into develop Aug 18, 2023
24 checks passed
@xmatthias xmatthias deleted the trades_data_handling branch August 18, 2023 16:04
@xmatthias xmatthias mentioned this pull request Sep 25, 2023
Open
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Enhancement Enhancements to the bot. Get lower priority than bugs by default.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@xmatthias @coveralls