-
Notifications
You must be signed in to change notification settings - Fork 78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rewrite the Profit & Loss calculation to persist it in the database #11
Comments
I've started to work on this in https://github.com/cryptoeax/arbbot/tree/pl-rewrite. Please note that running this branch before it's finished is dangerous if you don't know what you're doing. |
Yes, this is an example of what this rewrite is trying to solve. |
I do know why bot not sell 229.56399367 LBC/BTC on Poloniex, buy 229.56399367 LBC/BTC on Bittrex, but not sell on Poloniex. This is bug. If we have on two diffrent account on exchange total 500 LBC coin, after it we 729 LBC on two account on exchange and buy more more. I think that the total quantity of coins should not change and bot will need control it. |
@Scorpic1 you don't need to provide individual examples one by one, I am aware of the general issue. Like I said, this rewrite is about fixing the general problem. Right now the P&L section is an approximation. |
The bot always places limit orders, which means if the market moves too quickly beyond the price limit of your order, part or all of your orders may remain unfilled. This is an inherent risk of arbitrage. The bought could place a market order instead but that would be even riskier because then the trade could happen at any price. For example in the trade you are complaining about, you should be thankful that no LBC coins were sold, because what had happened was that the liquidity of the market at the limit price wasn't sufficient to satisfy your order, and had the bot placed a market order it might have had liquidated some LBC coins at a much cheaper price than desired. :-) Note that the APIs that the bot is using to talk to the exchanges are REST APIs so the bot isn't a high frequency trade bot and there can be hundreds of milliseconds of delay between the time we place an order and the time at which we look at the market order book. |
We can use the Websocket API no ? |
@cryptoeax Can we now use the pl-rewrite branch or we have to wait ? |
There two exchanges have no WebSocket API.
Not at all, please wait. When this work is ready I will merge it to the production branch. Matching the existing data against the trade history from the exchanges has turned much more difficult than I expected. I have something that mostly works now but is not yet good enough. The algorithm has a difficult time when there are several trades in a short window of time which haven't been completely filled. It currently sometimes picks the wrong trade for the wrong transaction. I have tweaked the algorithm quite a bit and I'm unsure how to make it much better. I may end up employing a different strategy and give up on importing the entire trade history from the exchanges, and just stick to what we have in the database and focus on getting the profit/loss information correctly after the transition to store things into the database. It's less ideal than my original intended approach, but it's not the end of the world, the future is bigger than the past... |
About WebSocket API. https://github.com/dparlevliet/node.bittrex.api |
This is completely off topic for the current issue, but let me elaborate. Bittrex has a WebSocket API that their own web UI uses but it is currently undocumented. Apparently they are going to release it in the next few months. The node client you have found is using parts of that API, but reverse engineering enough of what we need is a lot of work and it makes sense to wait for them to release proper documentation. Poloniex has some WebSocket APIs which we could use, specifically for the ticker and orderbook updates, but still no WebSocket API for actually placing the order so it would still not solve the problem I was referring to before. I have thought about using this API at some point in the future. |
@cryptoeax Can we test the functionality or do we still have to wait ? |
The work is almost finished, it's just one more thing that I need to implement before it's finished, but in fact if you are willing to run some manual mysql commands on your bot's database, the Here is how to test this:
The bot will prompt you for some things the first time it runs after the upgrade. Please pay close attention to the prompts and follow them carefully. When the bot starts to run and become operational again you should notice that most things are working like before. The one difference with the code right now is that the P&L UI will load faster (especially if your bot has made a lot of trades) but will not be kept up to date any more (that is the unfinished part) but don't worry about it, it's just a data presentation issue, all of your data is being stored safe and sound, and once you go back to the When you are done testing please run the following mysql commands:
Most of the changes made run during the first upgrade. I'm interested to know if you hit any errors there or after that while the bot is running. I'm also interested in feedback on the quality of the prompts, if you think they aren't clear, and if you have suggestions for improvements. Thanks! |
Some update fresh from the oven. I had some time to finish the work to calculate the profit/loss of each trade right after making one based on the information obtained from the two exchanges about how much actually was traded on each side which is the last big remaining piece. So right now if you test the One thing that is interesting to know while testing is that by going back to the I would appreciate if people can test this a bit. I'm planning to run this branch for a bit to see if I spot any problems with it that I haven't seen before. |
cryptoeax Thank you very much for your work! I want to start testing the |
Why do you not want to upload your trade history files? Going forward the bot will need those files to be able to run correctly. |
(Note that this experimental branch will be merged to |
Upload history files .csv
Have Error below. With 00:01:51: ARBITRATOR V2.0 launching... Fatal error: Uncaught exception 'Exception' with message '[POLONIEX] API error response: Nonce must be greater than 1513296113282620. You provided 1513296113282620.' in /var/www/arbitrage-bot/bot/xchange/Poloniex.php:573 |
We have seen this bug in many other issues. :-( I have been trying to figure it out but I haven't been able to reproduce it. Can you run the following command in |
root@h1685VM:/var/www/arbitrage-bot# ls -lR ./bot: ./bot/xchange: ./lib: ./web: ./web/flot: ./web/scripts: |
Generated NEW API key, error same. =( |
Thanks for posting this. This unfortunately shows the previous guess I had for why this happens isn't correct either. I will keep thinking. :-/ |
BTW you only need to place the CSV files in the root directory not |
In root have same files. I am delete from |
Yes, deleting those won't fix the bug, just thought to mention that the bot will never read the CSV files you have placed in |
For those following along, the problems we were discussing with the Poloniex API issues should all be fixed now. I have also pushed a whole bunch of new fixes to the |
cryptoeax thank you for your work! I am start testing After bot buy SYS on Bittrex and sell on Poloniex 23.53015660 SYS What bot waiting so long time about 30min. And do not write record about this orders in P&L.
What's happening? And have other problem re-balance worked is not correct and bot do not write info xfers window.
Why bot send all SC coin to Bittrex account? Why not half of all SC coins? Thanks. |
This was fixed in #33. |
With #47 fixed, the tip of the If you see any final problems or bugs on this branch that have not been fixed with the latest code update that I just pushed, please raise them now. Thanks! |
I'm planning to merge the pl-rewrite branch to production tomorrow unless if I find any last minute issues, and with that, this issue can be finally closed! |
I've merged the pl-rewrite branch into production and deleted the branch. If you were using the pl-rewrite branch, you can update the bot as follows:
If not, the next time you update to the tip of the production branch, the bot will prompt that it wants to update your database, and it needs your trade history files from the exchanges. It will tell you how to download and install them in the right place. I think I can call this fixed now. |
The current implementation of the P&L section is really a giant hack. Each time we want to satisfy a dynamic request from the web UI, we have to parse the entire log and look for trades. Even worse, the log will get deleted gradually, and its contents may be inaccurate due to past bugs and also in-flight transfers (since we currently use the asset balance difference before and after the trade to decide how much we have bought/sold so an incoming transfer can mislead us.)
I decided to spend the time to rewrite this properly. We need to persist the profit and loss information in the database so that we can quickly fetch it when needed. Also it would be nice to have a place where we have a record of each single transaction so that we can start to collect more information about transactions to see if we can come up with ideas on how to improve the bot. For example, I'm planning to record the order books at the times a trade is made to see if we could avoid incurring losses by detecting specific patterns in the order book...
There will be an import step require to import the current data into the database, which would be a one time operation. The bot will prompt about this on first launch after an update but otherwise these changes should remain invisible if everything goes successfully.
The way this will work is that we will try to match the information in the log against the order history data that we can get from the exchanges. Unfortunately the exchanges don't provide API access for the full order history, so the user will need to download the appropriate CSV files manually for this import process. I have a history of ~3200 transactions and the import code is now working well for this history...
The text was updated successfully, but these errors were encountered: