Graph Widget? Plot Widget?

[ccoupe]

Does anyone need a challenge ( @passenger94 ?). Long ago I did some some time series analysis foolery in Java and wrote a "widget" that would graph multiple time series data, auto scale the content to the window size, zoom in/out and move the zoomed graph left or right.

Hard core data analysis folks would just call out to gnuplot but gnuplot is huge and not really something we want to include in Shoes. Still, some cross platform visualization capability might be useful. I imagine, without thinking much on my part, that It would be Shoes widget (canvas) and we'd use mostly Cairo to draw and C/Ruby to get values out of Ruby arrays. Could it be done with pure Shoes/Ruby? Probably. Performance? I don't know - the test data sets I have are 5K of observations and they drew surprisingly quick on a Windows 98 laptop in Java 1.02,

[ccoupe]

Wow! That old Java code still works and just as painful to setup. Screen shot of graphing ^SPX (S&P 500) and ^IRX.R (3 month US Treasury Bill rate). The question is how to do it in Shoes and not what the plot tells you.

[ccoupe]

csv test data

[passenger94]

foolery in Java

sounds like a pleonasm :-D

haha, thanks for challenge :-), will look at it, if only for curiosity, i was interested in this when looking at the profiler thing and ways to get visual feedback
lack of time lately, some opportunities maybe next month

[dredknight]

this looks really awesome!
I have to get together and finish that popup thing and I can hop over and check this too :)

[ccoupe]

Still thinking out loud about this. It should be easy enough to convert that java code into a shoes.widget in pure ruby. Looking at the java, the DataSeries class is actually the bigger problem because I know it's sordid secrets - it has no way to represent 'no data' or 'bad data' for 'this observation' and it assumes all DataSeries have the same date/time starting point and ending point and number of observations. In order to be part of Shoes, the drawing code has to be in lib/shoes/ and while there are reasons to have DataSeries be a Gem, there are good (Loose Shoes symlink) reasons to put it into lib/shoes/DataSeries/

I've uploaded some Stock Market data files (csv format) to here - the contents are obvious. There is a long ago, mostly useless reason they start with '^' but it's what I have.

I've created branch 'graph' which is where this experiment should live.

[ccoupe]

Pure Shoes/ruby - test of concept. It works. Of course it's missing all kinds of things.

Good enough to try sub classing Shoes.widget.

[passenger94]

i say YES !!!
👠 👠 👠 🔆

[ccoupe]

So it's now a Shoes::Widget and it's time to add the axes and labels. OOPS! Shoes doesn't have methods for drawing text at x,y. Time to move c/obj-c? Maybe Shoes should have an Art Shape (like oval, star, etc ) named Art_String ??? so art_string {x:, y:, font:, fontsize:,color:, length: , text:}. I don't know how hard that is to do but it has some appeal and might be useful for other folks for other applications. Probably easier to do than creating a new c/obj-c graph widget so I like that. It might be useful text on top a image (with transforms already available like rotate and skew). Opinions?

[ccoupe]

Considering that I'm just converting my java code, it would be conceptually easier to create a widget,
similar to svg which I understand a lot better than Shape manipulation code (#266). It would have better performance for large data sets but not nearly as flexible for future enhancements. It's possible to do both a native widget and a Shoes::Widget version.

I think I'll call the native widget 'plot' unless someone has a better name. API is pretty simplistic so far:
pl = plot width, height. {options} and pl.add {options} | dataseries and pl.redraw in case someone is appending real time data. The option names: and defaults to be determined later but :data => [array of floatables]`, :name => "string", :minv => float, :maxv => float are the minimum.

[ccoupe]

API alert - some options belong to plot or chart (title and caption, width and height) and some belong to the data (like name, display_name, values and many more); The plot/chart behaves differently depending on how many arrays it's supposed to display. So, it's a two step creation process in Shoes plot width, height, {options) and plot.add {many_hash_args}

[ccoupe]

[Updated 2016-08-19]

[ccoupe]

Getting better. So many things to do under the covers.

The potential api keeps expanding with all the options the users might like ( turning of the grid lines for example)

[ccoupe]

The plot widget is almost good_enough for people to try it and find out what doesn't work for them. It does work on all platforms. I might think about including it in 3.3.2. Even if its a little half baked. Documenting it is going to be a challenge.

and the script to create that:

# good-graph.rb
Shoes.app width: 620, height: 610 do
  @values1 = [24, 22, 10, 13, 20, 8, 22]
  @x_axis1 = ['a','b','c','d','e','f', 'g']
  @values2 = [200, 150, 75, 125, 75, 50, 125]
  stack do
    para "Plot Widget Demo"
    flow do 
      button "quit" do Shoes.quit end
    end
    widget_width = 600
    widget_height = 400
    @grf = plot widget_width, widget_height, title:"My Graph", caption: 
      "Look at that! Booyah!!" , font: "Helvetica", auto_grid: true
    @grf.add num_obs: @values1.size, values: @values1, xobs: @x_axis1,
       name: "foobar", minv: 6, maxv: 26 , long_name: "foobar Yy"
    button "add #2" do
      @grf.add num_obs: @values2.size, values: @values2,
        name: "bartab", minv: @values2.min, maxv: @values2.max
    end
  end
end

[ccoupe]

As we often do, we seem to be talking past each other until we figure it out. Remember that chart creation is a two step process - create the widget (chart type) and then add data/labels to it. For line, bar, and timeseries, you can add multiple collections of 'Thing X' to the chart. Thing X also contains important things like the color you want and the min/max used to scale that part of the chart.

I really don't care what the Shoes object is named, I think it needs to exist and it does exist because the C was getting unmanageable and that wouldn't get better for new chart types or future enhancements to Thing X. From Tests/plot/gr3.rb

    @grf2.add values: @values1, labels: @x_axis1,
       name: "Bar", min: 0, max:  30, desc: "foobar Yy", color: "crimson",
       points: true, strokewidth: 12
    cs2 = chart_series values: @values2, labels: @x_axis2,
       name: "Tab", min: 50, max: 230, desc: "BarTab", color: "green",
       points: true, strokewidth: 6
    @grf2.add cs2

Both @grf.add 'hash' and 'chart_series' work. User's preference. The 'hash' version just creates the chart_series for you behind the scenes. Which one is the simplest? Minor differences, IMO and we have both, and always will. . Which one can be sub classed and re-purposed for other duty should someone who is not newbie prefer? - hash args in a Ruby program they cant change by a running script or an object they can build on?

Currently Thing X is called 'chart_series' at the C and Ruby levels. In my wandering around the interwebs I haven't found a common name for what Thing X is. I also haven't seen many FOSS Windows or OSX programs that draws simple charts, simply. Niche? I do care about doing the best thing given the situation we find ourselves in. We all do. Actually I prefer data_series and shoes_plot_data_series but that feels presumptive and invites a name clash with some ruby gem somewhere.

[IanTrudel]

@ccoupe thanks for explaining. Hash is simpler. ;)

By the way, does color: supports rgb() values? It should be noted that other Shoes methods support Shoes colornames, such as background, which does not use quotes as in Test/plot/gr3.rb. Consistency is important.

Hopefully you understand that I am not nitpicking. The chart feature is absolutely fantastic. I remember wanting to get some Java charting in the past but the only decent packages were available for sales. So it's refreshing to get it for Shoes.

[passenger94]

@backorder, yes i do agree, C functions should check for a ruby color (rgb or named) in the hash given to plot related methods (like texts, shapes, etc ... do for stroke, fill and those you mentioned, background, border ...).
The whole plot extravaganza is shiny enough now to spend time on killer "details" :-)

[ccoupe]

I welcome help and you are helping: Turns out the plot color can easily be handled as you suggest. Bonus points because I get to delete some ugly code. I wasn't fond of the string myself. Fixed.

Another annoyance is having to use strings for "dot" and "rect" - I'd prefer symbols :dot and :rect . Opinions?

The radar/spider chart ineed a lot help if someone has some extra time.

[IanTrudel]

Great!

Another annoyance is having to use strings for "dot" and "rect" - I'd prefer symbols :dot and :rect . Opinions?

Can't we have similar thing as Shoes colours? That would end up with dot and rect directly (without quotes or colon). Otherwise symbols would be fine as far as Ruby goes but it's always a bit strange to use with named parameters, such as points: :dot (a lot of colons if you notice).

The radar/spider chart ineed a lot help if someone has some extra time.

What's the TODO list on this one?

[ccoupe]

For better or worse, all of the color names are real Shoes objects in the Shoes name space. While we could do something similar and create Shoes classes for dot, box, rect, and circle, I am not comfortable that - it could to break existing scripts that have those class names. At the C level, I'm not happy to pollute Ruby space with the :symbols either but since "Shoes is not Ruby it only looks like it", symbols seem a better choice at the moment. (assuming I do anything)

What's the TODO list on this one?

Huge. I don't know a sin() from a cos(), radians from degrees, and when to use one vs the other. The code in there is wrong (converted from pycha). One of the magazines I read (Car and Driver) presents their results in a spider graphs (they also get complaints about them - but that's not our problem to solve). Radial axis can run from small number near the center to larger number on the edge. Or larger numbers at the center and smaller at the edge - depending on how they define goodness. - a reversal of current min/max - The code in there might handle that, might not - I suspect not. Then there is issue of labels. One of the charts in your example way above this comment appears to use axis in 0->100,000 but the labels are 50, 70, 90. Another reason for the &blk the chart_series/hash/Thing_X, on add.

[IanTrudel]

re: namespace

This is an important question. Creating additional Shoes objects would promote consistency and better syntax but your argument that it might break existing code is absolutely valid. It may not have big impact or may be expected from upgrading process but it's still important enough to consider before moving forward with such approach.

Ruby symbols would be an alternative and we could depreciate Shoes objects (such as colour names) via a warning in Shoes console to once again promote consistency. The depreciation would include support of colour name as symbols and we would phase out Shoes objects some future release.

This is something to think about because Shoes has been having increasingly new features since you took over the development. It comes to reason we need to have some kind of guideline.

re: math

Ruby Math (and most programming API) only support radians. One radian equals 1/2π. Degrees are something human enjoys and goes from 0 to 360 degrees for a full circle; exceeding values results in circling again, negative values for counterclockwise.

What you need to know is that 2π radian equals full circle (360 degrees). You use this reference to know how to split equally. You have X values on a radar chart? 2π / X.

Something along this line... not sure whether it is helpful to you...

[ccoupe]

For now, I'm going to leave points and a few others as strings args until I think harder about symbols and learn how to use them at the C level. I think I know and what I think is ok-maybe.

What you need to know is that 2π radian equals full circle (360 degrees). You use this reference to know how to split equally. You have X values on a radar chart? 2π / X.

That bit of knowledge helps. I 'll review the the pycha radar to C/cairo conversion with that extra info.

[ccoupe]

One use case for chart_series (aka Thing_X): Imagine you want to monitor the temperature of a probe in a BBQ smoker - when its running between set points of 225F and 240F you draw a black line/datapoint on the graph. If it drops below 225F then draw a blue line and if rises above 240F then draw a red line. Imagine it spikes up to 280 and you need rescale the y axis too. You'll need getters and setters for all the hash arguments for chart_series (plot.add) but if you don't have a backing object then those would be getters and setters on the plot object and each of them would require an index or hash arg to pick which series to modify. Unclean in C and unclean in Shoes.

See Test/plots/cstest.rb - it appends (label, value) but it can also replace existing (label, value) - just change that 7 to a 3 for example. Also note in gr4.rb that it uses the return object of a plot.add (currently a chart_series).

[IanTrudel]

So Shoes charting allow realtime data and other modifications?

[ccoupe]

So Shoes charting allow realtime data and other modifications?

One of my design goals. Fundamental. Not RT as process control people think but, every second or two.

[IanTrudel]

I'm looking forward to it. :D

[ccoupe]

Already there in gr1.rb which adds random data points every 5 seconds.

[ccoupe]

New beta available . Seems to mostly work Windows for graph tests but you need the graph test script which you only get from github sync. Perhaps my test scripts need to be in samples? expert-plot1.rb ...expert-plot6.rb ? Ick.

[ccoupe]

Who knew? Gruff is a ruby (rails gem) that draws all kinds of charts, including spider, using rmagick as the backend but at the Ruby level the code is pretty similar to what C/Cairo uses. Anyone attempting #266 charting package might want to start with this.

[IanTrudel]

It would have been nice but Gruff doesn't seem maintained anymore. The last commit is 5 months old and issues are dating in years.

Furthermore, rmagick as a backend might not be suitable for realtime graphs.

[ccoupe]

Gruff is source code to copy and modify ourselves. It's not too old and it does work within it's limits and they have test data, scripts and test output. It's hard to install (even on ubuntu and probably very difficult for windows). I would never use it as a gem to include in Shoes.

The algorithms inside are the value to me (and the test's, so I know that it should look like). I doubt anyone will modify the Ruby code to use Shoes shape drawing code in place of rmagick , but they could (#266).

[ccoupe]

We have radial axis.

[IanTrudel]

Good. I'd like to recommend a 90 degrees rotation to match other radar graphs samples we shown here (e.g. LibreOffice). You will certainly notice the starting point on your graph is on right-middle quadrant. LibreOffice et co. would normally start on top-middle quadrant.

[ccoupe]

I was working from the simpler spider in gruff which for whatever reason draws it with that start position. What we really want is Gruff's 'net' chart, which is a bit more complex but easier to comprehend than the pycha code.

[ccoupe]

This issue is getting a little slow to load and it won't get faster. I've opened #288 for radar/spider issues.

[IanTrudel]

I am not a huge fan of pie_percent on L019. How about just percent?

[ccoupe]

Closed in Shoes 3.3.2