Unable to filter out null objects if referenced by encoding

[LucidObscurity]

{
  "$schema": "https://vega.github.io/schema/vega-lite/v2.json",
  "description": "Unable to truly filter data",
  "data": {
    "values":[
      {
        "date": "2000-01-01"
      },
      {
        "date": "2000-02-01",
        "options": {
          "price": 16 
        }
      },
      {
        "date": "2000-03-01",
        "options": {
          "price": 17 
        }
      } 
    ]
  },
  "transform": [{"filter": "datum.options != null"}],
  "mark": "line",
  "encoding": {
    "x": {"field": "date", "type": "temporal", "axis": {"format": "%Y"}},
    "y": {"field": "options.price", "type": "quantitative"}
  }
}

This will throw "ERROR TypeError: Cannot read property 'price' of undefined" in the console.
I'm not sure why "options.price" would ever try to be evaluated for the record which was supposed to be filtered out.

[domoritz]

Should be addressed as part of #3369

[nyurik]

@LucidObscurity to work around, use compiled Vega panel at the bottom, copy/paste the Vega code into the editor, and move your filtering transform to the top of the list.

[domoritz]

Or add a calculate transform and flatten your data.

[nyurik]

@domoritz i think the input parsing happens before all of the transforms... Do you have an example?

[domoritz]

Vega-Lite will not generate a parse expression if you use a calculate.

[domoritz]

This works fine if you have an empty object. I don't know whether it is worth adding a lot of special code to support this case. It's pretty tricky to get right with selections and so on.

{
  "$schema": "https://vega.github.io/schema/vega-lite/v2.json",
  "description": "Unable to truly filter data",
  "data": {
    "values":[
      {
        "date": "2000-01-01",
        "options": {
        }
      },
      {
        "date": "2000-02-01",
        "options": {
          "price": 16 
        }
      },
      {
        "date": "2000-03-01",
        "options": {
          "price": 17 
        }
      } 
    ]
  },
  "transform": [{"filter": "datum.options != null"}],
  "mark": "line",
  "encoding": {
    "x": {"field": "date", "type": "temporal", "axis": {"format": "%Y"}},
    "y": {"field": "options.price", "type": "quantitative"}
  }
}

[domoritz]

However, maybe it's enough to get it working in format parse so that you can flatten your data afterwards.

[LucidObscurity]

Thanks for the follow up. I switched to the full Vega schema and the issue was resolved. I also resolved the reason why there were nulls in the first place, by building an API in between Kibana and Elasticsearch to generate my own statistics and add them to the JSON, but that issue exists in another repo. ;)

Thanks again, hopefully your follow up will help others.

[domoritz]

I have a fix in #3676

[LucidObscurity]

Nice. I do love the simplicity of Vega-Lite for most scenarios.

[domoritz]