Data has been down sampling when render with LineChart

[yangdong]

I am using LineChart to display data collected from somewhere every 5 seconds on a daily basis, which means my chart will be maximum about 17000+ points.

However, I found LineChart won't render the whole points, instead, it will down sample my data to about 2 minutes.

I've go through the MPAndroidChart docs and your examples, but no luck.
Can you please help me on this? here are my code:

// TODO: setup max zoom out value
// TODO: Long press to call out cusor, short for drap
import React, { Component } from "react";
import PropTypes from "prop-types";
import { AppRegistry, StyleSheet, Text, View, processColor } from 'react-native';
import { EasyDate } from "../../utils/date";
import themeVar from '../../theme/variables/common';
import { LineChart } from 'react-native-charts-wrapper';
import { HighLight } from "./highlight";
import _ from 'lodash';

export class IndicatorChart extends Component {
  state = {
    values: this.props.chartData.data,
  };

  handleSelect(event) {
    this.highlight.refresh(event.nativeEvent);
  }

  formData() {
    return _.merge({ dataSets: [{ values: this.state.values }] }, this.props.dataSetsConfig);
  }

  render() {
    const { chartData } = this.props;

    return (
      <View style={styles.container}>
        <HighLight ref={(highlight) => this.highlight = highlight }/>
        <LineChart
          style={styles.chart}
          data={this.formData()}
          chartDescription={{text: ''}}
          legend={this.props.legend}
          marker={this.props.marker}
          xAxis={{...this.props.xAxis, ...{
            axisMinimum: EasyDate.startOf(chartData.date).toNumber(),
            axisMaximum: EasyDate.endOf(chartData.date).toNumber(),
          }}}
          yAxis={{
            left: {axisMinimum: chartData.min},
            right: {enabled: false}
          }}
          drawGridBackground={false}
          drawBorders={false}

          maxVisibleValueCount={20000}
          touchEnabled={true}
          dragEnabled={true}
          scaleXEnabled={true}
          scaleYEnabled={false}
          doubleTapToZoomEnabled={false}
          dragDecelerationEnabled={true}

          autoScaleMinMaxEnabled={true}
          onSelect={this.handleSelect.bind(this)}
          onChange={(event) => console.log(event.nativeEvent)}
        />
      </View>
    );
  }
}

IndicatorChart.propTypes = {
  chartData: PropTypes.object.isRequired,
  legend: PropTypes.object,
  marker: PropTypes.object,
  xAxis: PropTypes.object,
  dataSetsConfig: PropTypes.object,
};

IndicatorChart.defaultProps = {
  legend: {
    enabled: false,
    textColor: processColor('blue'),
    textSize: 12,
    position: 'BELOW_CHART_RIGHT',
    form: 'SQUARE',
    formSize: 14,
    xEntrySpace: 10,
    yEntrySpace: 5,
    formToTextSpace: 5,
    wordWrapEnabled: true,
    maxSizePercent: 0.5,
    custom: {
      colors: [processColor('blue')],
      labels: ['Company X']
    }
  },
  marker: {
    enabled: false,
    digits: 2,
    markerColor: processColor(themeVar.toolbarDefaultBg),
    textColor: processColor(themeVar.primaryTextColor),
  },
  xAxis: {
    valueFormatter: 'date',
    axisMinimum: 0,
    axisMaximum: EasyDate.endOfToday().toNumber(),
    valueFormatterPattern: 'HH:mm',
    labelCount: 9,
    labelCountForce: true,
    textSize: themeVar.secondaryFontSize,
    position: 'BOTTOM',
  },
  dataSetsConfig: {
    dataSets: [{
      label: '',
      config: {
        lineWidth: 1,
        drawCircles: false,
        drawValues: false,
        highlightColor: processColor('black'),
        color: processColor(themeVar.brandPrimary),
        drawFilled: true,
        fillColor: processColor(themeVar.brandPrimary),
        fillAlpha: 60,
        valueTextSize: 15,
        valueFormatter: "##.00",
        mode: 'HORIZONTAL_BEZIER'
      }
    }]
  }
};

const styles = StyleSheet.create({
  container: {
    flex: 1,
  },
  chart: {
    flex: 1
  }
});

[wuxudong]

Can you add a log in render method to print size of data?

[yangdong]

@wuxudong thanks for you quick response.

the size of the is 12000+,

[yangdong]

you can tell from the two pics above that the time duration between them are more than 2 mins, but the original data is only 5 seconds duration.

[wuxudong]

What is the origin x value, not the time in 'marker'.

[yangdong]

it's the unix timestamp, like

currently, xaxis is defined as float, and it should be overflowed, is that the issue?

If so, how should I set x as time?

valueFormatter: 'date' is supported by MPAndroidCharts.

[wuxudong]

#160 may help.

[yangdong]

I see, thanks man. This solves my problem.

[yangdong]

@wuxudong after making x value smaller, all data has been rendered.

However, I think this is NOT the root cause of this issue, after chasing down the code.

First, We do have a date value formater which also has customized date pattern:

// formatting
        if (BridgeUtils.validate(propMap, ReadableType.String, "valueFormatter")) {
            String valueFormatter = propMap.getString("valueFormatter");

            if ("largeValue".equals(valueFormatter)) {
                axis.setValueFormatter(new LargeValueFormatter());
            } else if ("percent".equals(valueFormatter)) {
                axis.setValueFormatter(new PercentFormatter());
            } else if ("date".equals(valueFormatter)) {
                String valueFormatterPattern = propMap.getString("valueFormatterPattern");
                axis.setValueFormatter(new DateFormatter(valueFormatterPattern));
            } else {
                axis.setValueFormatter(new CustomFormatter(valueFormatter));
            }
        } else if (BridgeUtils.validate(propMap, ReadableType.Array, "valueFormatter")) {
            axis.setValueFormatter(new IndexAxisValueFormatter(BridgeUtils.convertToStringArray(propMap.getArray("valueFormatter"))));
        }

Second, DateFormatter should be able to format very very very... large number unix timestamp should NOT be a problem at all. float or long is way larger than timestamp.

public class DateFormatter implements IAxisValueFormatter, IValueFormatter {

    private DateFormat mFormat;

    public DateFormatter(String pattern) {
        mFormat = new SimpleDateFormat(pattern);
    }

    @Override
    public String getFormattedValue(float value, AxisBase yAxis) {
        return format((long) value);
    }

    @Override
    public String getFormattedValue(float value, Entry entry, int dataSetIndex, ViewPortHandler viewPortHandler) {
        return format((long) value);
    }

    private String format(long millis) {
        return mFormat.format(new Date(millis));
    }
}

[yangdong]

#160 is just a work around.

[wuxudong]

The main problem of using timestamp as x value is data is too sparse.

For example, there is an entry like {x:1517369866415, y: 1} '2018-01-31 11:37', and another entry {x:1517373466415, y: 2} '2018-01-31 12:37'，
the duration between two entries is one hour, but if you draw it on chart, you will find there are 3600000 blank points between those two entries, Just like the chart in #160

[yangdong]

no, as I said that data I use to render only have 5 seconds difference between each other.

such as:
[{x:1517328000000, y: 1}, {x:1517328005000, y: 1}, {x:1517328010000, y: 1}, {x:1517328015000, y: 1}, {x:1517328020000, y: 1}, {x:1517328025000, y: 1}, {x:1517328030000, y: 1}, {x:1517328035000, y: 1}]

[yangdong]

I am able to fix this locally with a new formater for time series data like this. Pull request has been made, can you please have a check ?

#193

[wuxudong]

sorry, I forget that yours is line chart.

It should be ok to use Unix time as x in line chart.

I have no idea why it looks like this.

Maybe you can use some static data to test it.

[yangdong]

I think I find the root cause.

The precision lost when converting a long to float or float to long,

@Override
    public String getFormattedValue(float value, AxisBase yAxis) {
        return format((long) value);
    }

I did test

@Test
    public void shouldFormatDate() {
        DateSinceFormatter formatter = new DateSinceFormatter("HH:mm:ss");
        Date date = new Date("Thu Jan 12 2017 10:23:45 GMT+0800");

        System.out.println(formatter.getFormattedValue(date.getTime()));
        System.out.println(formatter.getFormattedValue(date.getTime() + 3000));
        System.out.println(formatter.getFormattedValue(date.getTime() + 6000));
        System.out.println(formatter.getFormattedValue(date.getTime() + 9000));
        System.out.println(formatter.getFormattedValue(date.getTime() + 12000));
        System.out.println(formatter.getFormattedValue(date.getTime() + 15000));
    }
}

output, which is not expected, is

10:22:42
10:22:42
10:24:53
10:24:53
10:24:53
10:24:53

The precision lost only happens when the number is big, not for small ones, I guess that's why DateSinceFormater works (in the pull request).

[yangdong]

this should be a specific issue for Java, let me try another way, convert unix time for float in react native (javascript).

[yangdong]

no difference, still got this issue.

[wuxudong]

I check the code in MpAndroidChart, it use float as x index, that means directly using timestamp will definitely be overflowed or cause precision problem.

#160 is a workaround. if x is not sequential, you can use unix timestamp/1000/60 as x.

[yangdong]

I don't that will fix my issue.

1. DateFormatter will only accept unix timestamp, I can't make x to be unix timestamp/1000/60, otherwise it will format a bad date
2. If i convert x like the solution in CandleStickChart: set xAxis lable to show date,Show weird。 #160, I have to provide all entries for an entire day (my x axis will alway be 24 hours) which will reduce performance
3. time series data should be precise enough to be displayed, and in my case second matters. so floating and long converting seems not a quite good implementation, but it's beyond this repo.

[wuxudong]

can following code work?

data = [
  {x: Math.floor(1514308190539/1000), ts: 1514308190539, ...},
  {x: Math.floor(1514318190539/1000), ts: 1514318190539, ...},
  {x: Math.floor(1514328190539/1000), ts: 1514328190539, ...},
  {x: Math.floor(1514338190539/1000), ts: 1514338190539, ...},
]

xAxis = {
  valueFormatter: data.map(item => moment(item.ts).format('MM-DD HH:mm')),
  ...
}

after divide by 1000, it can be handled by int/float, and there is no need provide all entries for an entire day.

[wuxudong]

the date valueFormatter can help.