live update of the chart #264
Comments
|
There are a few things you can try:
Please let me know if that helps. |
|
mpf.plot(dfObj,...,block=False) or mpf.plot(dfObj,...,closefig=True) just does not work I recorded my screen and you can see I have to manually close the plotted window to let the app continue execution, YouTube video link below: |
|
Correct me if I'm wrong, but I think what you really want is not for the plot to close automatically, but for the plot to stay open but continually update. Something like this perhaps:
Please let me know if that is the type of thing are looking for. If not, please explain exactly how you want your plot to appear and how you want your plot to behave. Thank you. --Daniel |
|
Yes, but animation does not work, animation you have to decide an interval, this is based on event when "message received". so when new stock price come in, then update the chart, the new data is the trigger for update not the predefind interval. it seems to me the solution is to use figure with axes, you plot the chart initially with a figure, then you let app to continue execute, then when data arrived, you clear the existing figure(or axes) and draw the new figure inside the plot. but it could not continue the app after it executed mpf.plot statement, it plotted the chart and I have to close the chart window use my mouse, then it continue. Looks like need to have plot run in another thread, then update it when needed from the main thread. |
|
after search more, found that below code can close the chart and continue app execution: this is not what I want, I need to use the same plot, and update what's in side. I think use figure, and clear figure and redraw figure might work. |
|
Because I only have some snippets of your code, it's difficult for me to be very exact in what to do, however I would suggest doing something along these lines ... First, and most important, you must be able to get the Axes object into you BEFORE you call fig = mpf.figure()
myaxes = fig.add_subplot(1,1,1)Then def on_message(ws, message):
# code to parse message and add data to dfObj .....
myaxes.clear()
mpf.plot(dfObj, ax=myaxes, type='candle')NOTE: You must get the This is what I meant in the whole "Alternatively" section, the last section, of my original answer. Let me know who that works. If you can't get it to work, then if you can provide access to the rest of your code I may be able to help figure this out. |
|
After change to subplot and it started to work (see Youtube below), my guess the plt.pause(3) is the game changer step. Now, how do I add VWAP to the candlestick chart? The incoming data does have that number. (if you want to have the code, I can email to you or post here, but you need to go to https://alpaca.markets/ to get your key to run it, I am not sure if I give you my key will violate the contract, the account is free, but you need to put some money in to have the key code to access the data through ploygon) initialize plot: after on_message event triggered: When ever the console windows post messages, my keyboard focus goes there, not sure how to stop that. Also how to add VWAP or other indicators to the chart is my next research. I will study more docs. My candle is missing head and tales, any idea? |
|
Looks good! I'm glad you got it working. I will be offline for a few days. Will try to touch base next week. Good luck! |
|
Greatly appreciated your help. When ever the console windows post messages, my keyboard focus goes there, not sure how to stop that, Also how to add VWAP or other indicators to the chart is my next research. I will study more docs. My candle is missing head and tales, any idea? (my guess is post market) |
|
The Mouse is repositioning to the most preferred next operation. This is a windows feature. You might have enabled the Snap To option in Mouse Pointer Options. |
@gchudublin It may take a full hour, or two, to go carefully through all the documentation, but that will be time well spent to better understand the package. You may want to clone this repository and make sure you can actually run all the notebooks in the examples folder. After that, you may want to play around with tweaking the code in those notebooks to get a better sense of what you can do with this package. Regarding the strange look of your candles. It probably has something to do with your data. I suggest that you carefully examine the @centi1 Thank you for contributing a comment to this issue. Much appreciated! |
|
Hi Daniel, I'm struggling with the same issue, but after "showing" the plot with mpf.show(), then nothing else helps not to get stuck with the plot. I am slowly starting to think this might be an IDE issue - I'm currently running my script within pycharm within a virtual conda env. Do you have any suggestion? My code might be different, but the approach and final goal is exactly the same as discussed so far here. Only difference - I need to do mpf.show(), otherwise I don't get any chart. An alternative I was considering was placing the entire thing within a worker thread and not caring about it being stuck - but apparently it needs to be executed from within the main thread. PS: Greatly appreciate the library! Already managed to create an amazing live chart with my own custom indicators. Now the only issue is how to integrate this into the rest of my bigger script, without blocking it.. |
|
@nocciolate |
|
Hi Daniel, I used the MACD example you provided above and tweeked it a bit. But the overall logic is the same. Adding block=False (or True) to the plot() function doesn't change anything for me. Adding it to the show() function does. With block=True in the show() function the code gets stuck in the animation function until the plot is manually closed. With block=False - the animation function does not get executed even once. The plot also remains completely empty (does appear, but remains blank). I also tried setting plt.ion() in the beginning of the script - same result as when block=False in the show(). I also tried putting everything in a different thread, but get an error message that the plot should be started from the main thread. If you have any ideas, let me know. I should be able to identically apply any code-wise references to the MACD example since the logic is the same. Perhaps multiprocessing might be a solution for me. Found this and wondering how to apply it in my case: Have a nice evening. |
This makes sense. When
Try also specifically calling
Are you able to get the MACD example to work with no changes? I suggest you try that first. If you can get that to work, then you can slowly, one at a time, apply your tweeks to see which tweek brings out the issue. If you can't get the MACD example to work as is, or any of the other animation examples, as they are, then let's focus on that alone first. |
|
I believe I finally figured it out - it was a pycharm issue indeed. When running the script within the interactive python console within pycharm, then the plt.ion() works just fine. When using a while loop after the plot() I had again the same issue, but managed to overcome this as well by including the plt.pause() within that while loop. It's not the prettiest solution and I'm not sure where and how ofter I will have to paste this plt.pause() in the rest of the code, but at least it seems to work. If you have any suggestions how to do this in a more pythonic way, please let me know :) Thanks for your feedback! Here's my final solution so far: |
|
@nocciolate |
|
Hi Daniel, thank you for taking care of the mplfinance :) One last question - you wouldn't happen to have any recommendations for doing this in a better / more pythonic way, do you? Cheers |
I don't think I can recommend anything else. I am not familiar with the PyCharm environment. I also don't understand what the problem is or why the All the best. --Daniel |
|
Hi Daniel, so far the best explanation I have managed to find is the answer to this question: There are also some useful links in the comments to that answer. From what I understand it's not an IDE issue, but rather a python/mpl issue. A possible (and extremely ugly) solution is to have the code for the plot in a different script, which is to be launched by a thread as a subprocess. The thread will be blocked afterwards, but can be left as it is. I know this is probably the very opposite of good practice, but it's the only solution I have so far. I will try out themultiprocessing approach (link here) as a last resort. I will update this commect, should I find something better. Cheers |
|
@nocciolate I wonder also if another way to handle the issue is, instead of making calculations between animation updates, perhaps place the calculations inside the animation function. That way the calculations are between each animation update, but they are also part of the animation loop itself. Just throwing it out there as an idea. I can't say whether I think it's a good idea or not, because I don't really know or understand enough about the issue to say. But it popped into my head because essentially that's what I am doing in the MACD animation example; that is, the MACD calculations are included in the Over the next few weeks, as my time permits, I think I will explore this further. As I mentioned, the link you provided at least gives my something concrete to play with to better understanding the issue. Along those lines, I have some questions that may influence the approach I take when I investigate this further: Aare you working on windows? If so, do you have same issue if you just run it as a python script under command prompt or power shell? Whether threading as a subprocess (i.e. not true threading which is impossible in Python) will help, I can't say. It certainly seems a bit complicated but it may provide a lot of freedom to more cleanly do separate tasks during the animation, such as collecting new data, and processing new data, and then passing that to the animation subprocess. Still, I kind of wonder if maybe its simpler to put all that directly into the animation loop. Probably one of those things that you just have to try and see which approach works and keeps the code maintainable. |
|
Great, thanks for the feedback. I have to think about it because it's a bit complicated. My problem is that I have a constellation where I am pulling live data directly from an exchange via two separate websockets. The first websocket pulls best bids in real time for over 300+ pairs and compares them to initially pre-recorded prices, while potentially executing orders if conditions are met (done similarly to the "on_message" logic in the conversation above). The second websocket pulls the exchange's responses from the persistence layer which are sent after an executed order of mine, while also potentially executing further orders. I have managed to tune it to run smoothly. I don't know in advance which pair I will be interested in. The first websocket needs to catch this and enter the market. After entering the market, it is supposed to "however" open the plot and shut itself down. Opening the plot includes firing up a new websocket for last executed orders in real time (I basically want a chart showing in real time prices and volume of market buy vs market sell orders). The plot is supposed to run in parallel, while not interrupting the second websocket (listening for responses to orders of mine) which would still be running. When I took it to "production" - basically tested in real life, I noticed that after the first websocket launches the plot (by calling the second script with the plotting functions, incl. the websocket for executed orders), it gets blocked afterwards.. which also blocks the second websocket... which makes the entire exercise pointless. I'm just extremely surprised this is so difficult to achieve, but time and motivation to overcome this are present. Probably just lacking skills :) PS: I might just create two identical copies of the script - one for trading, one for plotting, and pin them to two separate cores in order not to affect latency of the trading one. Have a great weekend |
That's almost exactly what I was thinking as I was reading your description before the "P.S." My inclination, however, is not to make two copies of the script, but rather split it in two: running two separate processes that do not interfere with each other:
Trade execution is mission critical, and that process should be kept as simple as possible, focussing only on trade execution and nothing else. That way, if something goes wrong (with the trade execution process) it will be easier to debug (and being simpler it will be easier to test, and less likely to have a bug in the first place). In other words, the trade excution part of the application should be kept as simple, independent, and decoupled from the rest of the application as possible. As an added benefit to this independence, it can be tested independently from the rest of the application too. You may want some kind of ipc between the executor and the analyzer, but only to tell the trade executor process to go ahead and execute a trade, and maybe have it send its info back to the analyzer/plotter (if needed), perhaps asynchronously so neither interferes with the other. |
|
Hi Daniel, thanks a lot for the advice - that was roughly also what I had in mind! I tested the combo yesterday on live market data and at least this part worked fine. There were some other issues with speed of processing and plotting, but it is outside of the scope of the topic here. May I ask you one more thing related to plotting live market data? |
|
@nocciolate |
|
My final goal is to have one live plot including 2 figures with 3 main components: a) OHLC chart in fig 1, b) volume split of market buy and sell orders in fig 2 and c) vizualization of the order book data as a heatmap behind the OHLC prices (again in fig 1). So far, with your help, I have managed to achieve a) and b) in python and they more or less work for periods of time, when there isn't much incoming volume to be processed. Sadly, I'm interested exactly in the opposite market conditions - but I will figure it out eventually. For c) I was imagining something similar to the service that bookmap offers. There's also a javascript lib, which I found to be extremely well done (link). From your experience - do you think python is the right way to go in order to achieve all three points, while the evnironment remains stable and suffiently fast? I am slowly starting to think javascript might be superior, when it comes to real time plotting of live market data. |
|
@nocciolate
The decision, for me, often comes down to how fast the code needs to be. Let's say I need a half-dozen plots, and even if they need live updates once per second, or once every two seconds, the speed difference is negligable; you probably would not even notice it under most conditions on most machines. In such a case, for me, the ease and beauty of banging out the code in Python far outweighs speed difference which, at those rates, is not noticible. (But if I need many updates per second, and/or hundreds of plots in a short amount of time, then speed may become an issue). Even where speed is an issue, depending on the application, it may be worthwhile to multi-process Python (again because of the ease of writing the code). I once wrote a market analysis program running some 20,000 different scenarios. I wrote the code in C++ and it took 2 to 3 minutes to run. It took me about a week to get the code fully functional. I then re-wrote the code in Python in just a few hours. Admittedly some of that writing speed (a few hours vs a week) was because I already knew what I wanted the code to do, but for sure most of it was because I did not have to hand-craft and unit test a lot of different classes. In fact, I did not simply rewrite the code from C++ to Python (keeping the same structure and design) which I imagine would have taken me at least two to three days. Instead, I used existing python classes and methods from pandas, numpy, and the standard Python library modules. I finished coding in about 3 hours. But the Python version took 55 minutes to run! The 20,000 scenarios were independent from each other, they could be run in any order and in parallel. So I used a python multi-processing module, running 8 parallel process on an 8 core pc, and got the time down to 6 minutes. For me, for that particular application, it was worth the extra time (6 minutes per run, vs 2.5 minutes per run) for the benefit of less code that was easier to maintain (because my goal was constantly tweaking the algorithm and re-running the 20,000 scenarios to see if I got better results). It seems to me however, that for a market data visualization to plot 3 or more updates per second, if that rate of plotting speed is important to you, then Python may not be the way to go. There are definitely ways to optimize the Python to handle a faster rate of screen refresh (compared to what I am doing in mplfinance) however, imho, once you are putting that much effort into it, you may be better using a faster language. Of course, to keep the code easy to maintain you may have to take the time to hand-craft and organize the behavior of a number of classes. So that's my take on it. Although matplotlib can plot heat maps (examples here and here), I have no intention to add heat maps to mplfinance in the near future; certainly not this year unless someone else volunteers to contribute the code. Now just to get a little philosophical about the market, i personally have never felt a need to see a visualization update more than once every two seconds or so. It seems to me that any decisions I am making based on the visualization are not going to be made that quickly. If I wanted to make decisions that quickly I would probably want the software to make the decision for me. I understand that other people find a lot of value in rapid updates to the data visualization. That's fine and good, and it appears to work well for many people. My own philosophy is that these differences (different people taking different approaches) are a significant part of what makes the free market work well. If everyone always did the same thing, and saw things the same way, imho, that would create friction and other problems in the marketplace. There is great value to diversity of thought and diversity of approach, in that they provide (among other things) opportunities for everyone. HTH. All the best. |
|
Hi Daniel, I highly appreciate you taking the time to go into such details in your reply! I agree with everything you said - in a normal case, which one already undestands well and can build a logic around, I would also always prefer a machine to be doing split second decisions instead of a human. On the other hand, I am still in a sort of "exploratory" mode and want to first understand what exactly happens within those 3-4 seconds I am interested in, before diving into automation. From all the trade-by-trade analysis so far, I believe to have the full picture of what happens when it comes down to how market orders are coming in and managed to develop a logic (incl. location from which the script is running) that gives me a certain advantage for early entry. In terms of speed of entry, I did manage to have a python script with a "loop" run-time of roughly 0.5 microseconds and an average of about 22-25 ms time between signal being "produced" within the previous auction block in the matching engine of the exchange (from the previous trades) and my entry order being successfully executed back in the matching engine. The exchange is running on aws and I'm sitting in the same server cluster, plus I have a certain logic for dealing with load balancers at the point of entry. Because run-time is so negligible compared to latency, the additional benefit from a different language would be relatively small, plus I'm already a bit familiar with python but not js or C++, I was secretly hoping I'd be able to do the plotting part (separate script) also in python, without having to start from scratch again and learn a new language. Once in the market - then the interesting part begins. Since the exchange does not provide historical data for limit orders (at least to retail traders), it's hard to analyse at a later point the action that occured in the order book post my market entry. I therefore wanted to plot the order book for a couple of times (for those 3-4 secs post entry) and come up with an additional exit logic for how certain conditions are to be handled - automatically as you also suggested. Given all you said, I believe I have two options - either switch to a different language or perhaps try plotting the order book volume through matplotlib on top of the ohlc bar chart. I will probably start with the latter. Just need to figure out if that's even possible. So far I couldn't figure out how to combine two different plots on top of each other. Best regards, |
|
Georgi, Regarding
Take a look at this: Acessing mplfinance Figure and Axes objects and the links therein. In the Keep in mind that, at least for the panels approach, it is assume that all plots have same datetime x-axis data. That is, they are all time-series data sets over the same period of time. A few other pages that may be of interest to you, in helping you decide how to accomplish what you want to do:
If you have not already seen the above, it may be well worth the time to allot about 25 minutes of your time to carefully read through the above examples. All the best. --Daniel |
|
Hi Daniel, thanks a lot for the links! I took the time to study the examples and have a much better understanding now what's going on under the hood. Yet, I'm still not sure how to achieve what I set out to do. I came to the conclusion I might need external help :) Thanks again for all your time and effort to help! I really do appreciate it. It has been a pleasure to talk to someone who understands you and you don't have to explain every little detail. Have a great day and take care! Best regards, |
I like to get around this too. How do I get live updates plotted on mplfinance, with reference to your diagram above? |
|
@Olujideo |
|
Hi Daniel,
Thanks
This is close to what I need.
…On Sun, Nov 21, 2021, 02:39 Daniel Goldfarb ***@***.***> wrote:
@Olujideo <https://github.com/Olujideo>
I am not completely sure what you are asking. The code for the example
shown in the plot that you referenced above can be found here:
https://github.com/matplotlib/mplfinance/blob/master/examples/mpf_animation_growingcandle.py
Let me know if that helps.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#264 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APRS3R2K5VZZ3HYCVRPTN63UNBEWFANCNFSM4RPLRHNA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
|
Binance live running candle stick chart of btcusdt import requests
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from mpl_finance import candlestick_ohlc
import matplotlib.dates as mpl_dates
from matplotlib.animation import FuncAnimation
url = 'https://api.binance.com/api/v3/klines?symbol=BTCUSDT&interval=1m'
headers = {
'Content-Type': 'application/json'
}
fig, ax = plt.subplots()
def update_chart(i):
response = requests.get(url, headers=headers)
data = response.json()
df = pd.DataFrame(data, columns=['timestamp', 'open', 'high', 'low', 'close', 'volume', 'close_time',
'quote_asset_volume', 'number_of_trades', 'taker_buy_base_asset_volume',
'taker_buy_quote_asset_volume', 'ignore'])
df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
df['timestamp'] = df['timestamp'].apply(mpl_dates.date2num)
df = df.astype(float)
ax.clear()
candlestick_ohlc(ax, df.values, width=0.0005, colorup='green', colordown='red')
ax.xaxis_date()
plt.xticks(rotation=45)
plt.title('BTCUSDT Candlestick Chart')
plt.xlabel('Date')
plt.ylabel('Price')
ani = FuncAnimation(fig, update_chart, interval=10)
plt.show() |
|
FYI, after mpf.plot, add the following work for me: fig.canvas.draw_idle() |
I have data streaming through API calls to Alpaca and it is real time stock market data, it is using the code below to get data, and "on_message" event triggers I parse the data to pandas dataframe object dfObj, then plot the candlestick chart. Now the issue is the chart will need to be closed manually in order for it to continue execute the next "on message event", anyway to update the chart? and continue without plotting a new one?
the code above just fragments
The text was updated successfully, but these errors were encountered: