Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Unified CSV #223

Merged
merged 27 commits into from Mar 21, 2024
Merged

Use Unified CSV #223

merged 27 commits into from Mar 21, 2024

Conversation

macanudo527
Copy link
Collaborator

This addresses several bugs. I tried to each part in a separate commit so that it is easy to review.

It basically fixes the problem with Kraken CSV files being put in one big CSV file.

I have to do a full pull to test it again. Then, I can fix the commented out parts in the CCXT pair plugin.

@macanudo527 macanudo527 self-assigned this Jan 31, 2024
@macanudo527 macanudo527 force-pushed the use_unified_csv branch 2 times, most recently from 7b63e51 to 2ea8c47 Compare January 31, 2024 13:27
@macanudo527 macanudo527 marked this pull request as ready for review February 1, 2024 00:41
@macanudo527
Copy link
Collaborator Author

Unfortunately I can't test this. Exchangerate.host has restricted their free tier to just 100 calls a month! Definitely need to find a better solution. There are plenty of places that allow you to download daily data for free in CSV form, but you need to register and download. User will need to manually download the CSV and then we can chunk it.

@eprbell
Copy link
Owner

eprbell commented Feb 1, 2024

Unfortunately I can't test this. Exchangerate.host has restricted their free tier to just 100 calls a month! Definitely need to find a better solution. There are plenty of places that allow you to download daily data for free in CSV form, but you need to register and download. User will need to manually download the CSV and then we can chunk it.

Oh, no, what a pain... Does the fact that you can't test it mean that we need to abandon this PR? If there's no better alternative, I think it's OK to ask users to download the CSV as needed.

@macanudo527
Copy link
Collaborator Author

Oh, no, what a pain... Does the fact that you can't test it mean that we need to abandon this PR? If there's no better alternative, I think it's OK to ask users to download the CSV as needed.

This PR should be okay. It worked before I did a code clean. I just wanted a final test after the code clean to verify everything. The free tier offers up 100 requests which might be adequate for people for now. That would include about 90 or so days of trades. I think for a normal user that will be kind of okay. I just run bots that are constantly trading, so I need every day of rates.

@macanudo527
Copy link
Collaborator Author

@eprbell We should have just use this from the beginning. -> forex-python. I'm not sure why it didn't come up before.

@eprbell
Copy link
Owner

eprbell commented Feb 2, 2024

OK, I'll review it then. Also I took a quick look at forex-python: it does look like what we need!

Copy link
Owner

@eprbell eprbell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay! Here's a first review. Thanks for the hard work!

README.dev.md Outdated Show resolved Hide resolved
src/dali/abstract_pair_converter_plugin.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/ccxt.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/ccxt.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/stubs/progressbar/__init__.pyi Show resolved Hide resolved
tests/test_plugin_ccxt.py Show resolved Hide resolved
@macanudo527
Copy link
Collaborator Author

@eprbell just an FYI. When I create [WIP] commit, I'll later force push a change to finish it up. I basically code on two different PCs and so I use Github to transfer changes between the two. Sometimes I can't completely finish a commit though.

@macanudo527
Copy link
Collaborator Author

@eprbell I'm not sure why the tests are failing. Pylint seems to have a problem with stubs.

Otherwise, this should be ready to go.

@macanudo527
Copy link
Collaborator Author

@eprbell do I need to do more for this? I'm not getting these errors locally.

@eprbell
Copy link
Owner

eprbell commented Mar 3, 2024

@eprbell do I need to do more for this? I'm not getting these errors locally.

This may be a bug in the latest release of pylint (https://github.com/pylint-dev/pylint/releases): it may be worth trying again after fixing the release to 3.0.4 or 3.0.3.

Copy link
Owner

@eprbell eprbell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's another round.

docs/configuration_file.md Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
@macanudo527
Copy link
Collaborator Author

This may be a bug in the latest release of pylint (https://github.com/pylint-dev/pylint/releases): it may be worth trying again after fixing the release to 3.0.4 or 3.0.3.

I set it to ignore all .pyi files. We were basically doing that before by declaring the directories in the cmdline.

macanudo527 and others added 2 commits March 4, 2024 11:01
Co-authored-by: eprbell <77937475+eprbell@users.noreply.github.com>
@macanudo527
Copy link
Collaborator Author

@eprbell Is there something special I need to do for the document check? That Kraken link isn't dead.

@eprbell
Copy link
Owner

eprbell commented Mar 4, 2024

@eprbell Is there something special I need to do for the document check? That Kraken link isn't dead.

That sometimes happen in Github CI (for mysterious reasons): the solution is to enclose the link in <!-- markdown-link-check-disable -->...<!-- markdown-link-check-enable --> so that it's not checked.

@macanudo527
Copy link
Collaborator Author

@eprbell is it possible to get this approved?

@eprbell
Copy link
Owner

eprbell commented Mar 7, 2024

@eprbell is it possible to get this approved?

Yes, I'll do one more full review in the weekend. I've been snowed under at work and didn't have much time.

Copy link
Owner

@eprbell eprbell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is one last big review from scratch: the comments are all fairly minor. Good work!

src/dali/abstract_pair_converter_plugin.py Show resolved Hide resolved
src/dali/abstract_pair_converter_plugin.py Show resolved Hide resolved
src/dali/plugin/pair_converter/ccxt.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
src/dali/plugin/pair_converter/csv/kraken.py Outdated Show resolved Hide resolved
@macanudo527
Copy link
Collaborator Author

@eprbell I removed the unified CSV id.

@eprbell
Copy link
Owner

eprbell commented Mar 19, 2024

Sounds good: as soon as other comments are addressed, I'll review one more time.

@macanudo527
Copy link
Collaborator Author

macanudo527 commented Mar 19, 2024

Sounds good: as soon as other comments are addressed, I'll review one more time.

@eprbell Oops, I committed the clarify comments commit to the forex PR. I cherry picked it to this one.

Copy link
Owner

@eprbell eprbell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: thanks for all the work!

@eprbell eprbell merged commit 2096a7c into eprbell:main Mar 21, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants