[WIP] draw indicator results in the UI

[askmike]

implements enhancement described in #690.

[Leonix]

I tried today to actually draw the results on screen. I managed to pass indicatorResults to plotter in chart4.js and then started to explore its contents. I figured indicatorResults has a structure similar to this:

{
    macd: [ {
        date: "2017-07-22T10:30:00.000Z",
        result: -4.417888845745903
    }, ... ]
}

This is not enough :(

Different indicators should be drawn differently: some on main canvas with the same y-axis as candles (e.g. SMA, EMA). Some on a separate canvas as a line chart (e.g. CCI, RSI). And things like MACD should ideally be two lines and a bar chart on its separate canvas, which requires more than one set of data points.

What I'm saying is that strategy should somehow define and communicate to the plotter how to draw its indicators. What @tomih suggested might be a good starting point.

[metachris]

This is a very good feature and I was wondering about the status of this PR. Would be great to merge the indicators for the UI.

I've been playing around with Highcharts/Highstock. It's extremely easy to use, has a very nice UI and is very flexible (just a few lines of code to add additional series, flags, etc.).

If we could merge this pull request, I would be interested in contributing another backtest view which uses Highcharts, which may make it easier to refine strategies and parameters.

See this screenshot for inspiration: ;)

[askmike]

This PR was only a start: it only has code that exposes indicator results via the API. There is no UI code that generates different charts per indicator yet (also needs a rebase with current develop branch).

I've been playing around with Highcharts/Highstock

I am very reluctant to ever adding highcharts to Gekko for licensing reasons. Gekko (and all it's dependencies) are entirely open source and some people (including me) are building on top of Gekko to create closed source applications (something which would become expensive by using highcharts). Nevertheless we could always expose whatever UI functionality we want to add as external software, here for example.

[metachris]

It could be possible to use Highcharts in the open source version and either remove it from the commercial products or to just license it. But I understand your hesitation.

For experimentation I implemented the indicatorResult from this pull request and a similar strategyResult, and added Highcharts to more easily inspect the data. I just want to quickly show off a little -- this is how it looks like at the moment:

[askmike]

Oh damn that chart does look very good...

[silentrob]

What is the status of this PR. I was just about to do the same thing with http://techanjs.org/ which has a more liberal license and we already have D3 as a dependency.

[askmike]

Hey @silentrob this PR thus far only includes 30694cd which exposes indicator results to the frontend (when backtesting). I won't have time to pick this up soon unfortunately, feel free to have a go at it :)

[chadillac81]

Something to consider with the the candlestick chart: 1) add a small countdown showing how long until next candle, 2) let the last candle update live, so you see the candle form as trade data comes in.

[askmike]

Hey @chadillac81

1. add a small countdown showing how long until next candle

Gekko calculates candles based on trades, not based on time. And since it's hard to predict when the next trade will come it's also hard to predict when the next candle will be. See #773 for details.

2. let the last candle update live, so you see the candle form as trade data comes in.

This would require a very different data model: your strategy only sees candles and the UI is supposed to reflect the data your strategy has. Such realtime functionality unfortunately doesn't fit in the datamodel (see here).

[chadillac81]

I see. Seemed like a cool idea, so I figured it pitch it :)

[xerix32]

Hi @metachris , can you share "Highcharts" code?
I do love to use it on my local installation... :)

[CptanPanic]

in lieu of this feature, as I do need to see the values of the indicators, how can I get the values of the indicators so I can plot them outside of gekko?

[askmike]

@CptanPanic I made start with this commit: f08e358

note that I am not sure it still works..

[CptanPanic]

@askmike how about if I just wanted to do it low-fi and just log all the indicator values to a csv file. Where would the best place to do this?

[erice1984]

I don't know if anyone saw this. I just found it yesterday. Maybe this could help you guys out.

https://www.tradingview.com/HTML5-stock-forex-bitcoin-charting-library/

[bbjay]

@erice1984 thanks, but their license allows to use it only on public websites, not personal/private ones. so its not suited for us.

[bbjay]

I've started an implementation with TechanJS, my code is in this branch:
https://github.com/bbjay/gekko/tree/feature/plot-indicators

This is how it looks so far with the DEMA example config:

Features so far:

• candles and volume plots
• plot overlay indicators with configurable color
• shows indicator and candle values on crosshair move

Planned next steps:

• replace techan.js macd & rsi subplots with configurable ones from strategy
• figure out a way to send multiple values from one indicator (e.g. ema1 and ema2 from DEMA) to the GUI, without recomputing them

Any help or input is welcome!

[mkondel]

@bbjay I downloaded your repo and checked out branch feature/plot-indicators. The plot from backtesting still looks the same, no candlesticks and just the green/red dot indicator. Could you post the steps you use to get this to work please?

[bbjay]

@mkondel the UI is not recompiled in this branch, so you need to set up and run gekko in UI dev mode:
https://gekko.wizb.it/docs/internals/gekko_ui.html#Developing-for-the-Gekko-UI-frontend

[nissimpradhan]

I did manage to pull @metachris's project onto the current gekko version and made it work.

@askmike Hi, Mike Sorry to ask this very simple question but i'm new to vue.js and i want to know why i'm not able to see the outputs from console.log that was put in .vue files.
I'm running
node gekko -ui
as well as
npm run dev
on folder web/vue

[nissimpradhan]

@askmike oh never mind, it's in dev tools in chrome.

[christianbaier]

@nissimpradhan hey would you share the code?
since the repo from @metachris is deleted

[metachris]

Oh dang, just wanted to cleanup my repos. I can push my old branch to any repo, just add me as collaborator and let me know

…

On Jan 25, 2018 17:16, "Christian Baier" ***@***.***> wrote: @nissimpradhan <https://github.com/nissimpradhan> hey would you share the code? since the repo from @metachris <https://github.com/metachris> is deleted — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#793 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHIyyK2WZ0uCaGI3NC_FyZ4J2CFUL_Jks5tOKjwgaJpZM4OAOVw> .

[christianbaier]

@metachris im kinda new in the github world i think i added you ;)

[metachris]

https://github.com/christianbaier/gekko/tree/feature-backtest-datapreview

[ansonphong]

I am very interested in this feature. It would be great to have the option to register a line to draw on the main graph as well, and change the line parameters.

[nissimpradhan]

@ansonphong yeah this feature can be implemented on metachris's branch quite easily, which i have done.

[Jabbaxx]

@nissimpradhan I am quite interested in that topic. Would you mind to share your work us :) ?

[nissimpradhan]

You should check out the branch that metachris has. My work is only a few lines on top of that. The code for drawing new lines on high charts are on a backtestResultChart.vue.
There's variables to extract data from any indicator already. You just need to plug them into a line and create it.

[Jabbaxx]

@nissimpradhan
Thanks a lot :)

Just to make it easier for the next who wants to try ;)
To add for instance the obv TALIB indicators. Add it in web\vue\src\components\backtester\result\backtestResultChart.vue
res.result.outReal is the data from the TALIB indicator. Address it just as you do in your strategy.js
Here one new line:

const indicatorResultMapFunctions = {
'talib-dema': (res) => [moment(res.date).unix() * 1000, res.result.outReal],
'talib-obv': (res) => [moment(res.date).unix() * 1000, res.result.outReal],
default: (res) => [moment(res.date).unix() * 1000, res.result]

and one text further down if you want it on the secondary chart:

const secondaryAxisIndicatorTypes = ['DEMA', 'CCI', 'talib-obv'];

Have fun with it.

[Jabbaxx]

@nissimpradhan I have looked into that highchart edition and it works very nice :)

As I can see highchart also supports a candle stick chart https://www.highcharts.com/stock/demo/candlestick

But it looks like we only have the Close price in the candles, Open High and Low are all zero if I do like this:
// data: this.result.candles.map((candle) => [moment(candle.start).unix() * 1000, candle.close]),
data: this.result.candles.map((candle) => [moment(candle.start).unix() * 1000, candle.open, candle.high, candle.low, candle.close]),

And I am a little to inexperienced to find out how to get them there. Do you have any idea ?

[shufps]

Is there any progression?

Drawing something/indicators in the chart is a MUST feature for backtesting ... I can't believe, it hasn't been implemented in the official release yet ... For instance, how can you fine-tune an e.g. EMA which contributes to triggering buy and sell decision if you can't even check, where the problems are when backtesting ...

[nissimpradhan]

@Jabbaxx I just tried this for candlestick, it works for me. Let me know if this works.

series: [{
       name: `${this.result.report.asset}-${this.result.report.currency}`,
       id: 'trades',
       data: this.result.candles.map((candle) => [moment(candle.start).unix() * 1000, candle.open, candle.high, candle.low, candle.close]),
       color: colors[0],
       type: 'candlestick'
     }

@shufps Yes this feature is not there in the official version because of some licensing issue apparently. If u scroll up top u can see this comment about it from the dev

[Jabbaxx]

@nissimpradhan Thanks for answering :)
Unfortunatly not :( it just looks like this, no line and no candles. This was also what i saw in my tests.
Did you change any other things and what version of gekko you are running ?

[nissimpradhan]

I'm running the version 0.5.13, with metachris's branch merged to it.
Did you try importing both,
import * as Highcharts from 'highcharts/highstock'
import * as HighchartsMore from 'highcharts/highcharts-more'
Also did you build the file from

[dzham09]

to fix the candle, modify backtester.vue, add the open, low, and high to the candleProps and rebuild. Example below.
const req = {
gekkoConfig: this.config,
data: {
candleProps: ['close', 'start', 'open', 'low', 'high'],
indicatorResults: true,
strategyResults: true,
report: true,
roundtrips: true,
trades: true
}
}

[Jabbaxx]

@dzham09 Geniously... :) That was what was missing :)
In combination with @nissimpradhan tip, it works :)

Thanks a lot both of you :)
Keep an eye on the #610 thread. I will one of the next days post a newer version of Algotester.py for backtesting a lot of combinations of parameters. It uses the backtesting of gekko. And now uses the roundtrip values to sum up the result.. The one i posted last time is still buggy.

[Jabbaxx]

Has anyone of you looked into what strategyResults: true delivers ?
Is it worth looking at ?

[dzham09]

I tried for some time to figure out why strategyResults do not show any value. backtestHandler.js is the entry point to the chart fetching the data, but it never fires the strategyResult message. I noticed also that the indicators must have values within a range so some indicators will not show.

[Jabbaxx]

OK. Maybe I will look into it later.
If you want red/green candles ;) then put this in the top where "colors is defined:
const rgcolors = ["#c94141", "#90ed7d"]

And add "upColor":

     series: [{
	   name: `${this.result.report.asset}-${this.result.report.currency}`,
	   id: 'trades',
	   data: this.result.candles.map((candle) => [moment(candle.start).unix() * 1000, candle.open, candle.high, candle.low, candle.close]),
	   color: rgcolors[0],
       upColor: rgcolors[1],
	   type: 'candlestick'
	 }
	 }],

would be nice if we could make a volume bar in color, but looks like very difficult. Managed only in one color ;)

[Jabbaxx]

Another small tip ;)
Put the flags on top of the candle HIGH value. That way you can read the candles better:
add : onKey: 'high'

Like here:

const flagSeriesId = `f${chartOptions.yAxis.length}`;
chartOptions.series.push({
  type: 'flags',
  name: 'Flags',
  id: flagSeriesId,
  data: buySellFlags,
  onSeries: 'trades',
  onKey: 'high',

[askmike]

Unfortunately this PR is outdated, we still want the same behaviour but it will be implemented differently. All changes in this PR are already part of it's successor: e7d7e56

We hope to launch that asap!

[arabaey5566]

@bbjay
Did you continue your implementation with TechanJS ? I would like to use it also...

Could you explain me in detail what needs to be done ? What files to update? Compile again? How?... I'm new to node.je and gecko.
Thanks, Anthony

[jurgenbosch]

Any progress on the update?

[mboremski]

will this be implemented in develop someday??
looking forward to it a looooong time.. :)

[verbraucher]

I think this feature is already part in the gekko Quasar UI:
https://github.com/H256/gekko-quasar-ui