r.what: Add JSON output

[kritibirda26]

Added an option format which accepts the value "plain" for the current output format and "json" for the JSON output. The newly added parson library is being used here.

Also, added a couple of python tests to ensure JSON output works as expected.

fixes #3518

[echoix]

I think you can ignore (resolve) the Clang-Format suggestions that are already marked as outdated (since you fixed them before they got published, they were queued from other CI activity in the OSGeo organization)

[kritibirda26]

Thanks for the tip, @echoix.

[echoix]

Is the way the tests written testing the json output, or python's json parsing?

[kritibirda26]

The JSON is output by the r.what tool and then it loaded by using python's json module. Python's json module loading the output of the r.what tool successfully shows that the output was valid JSON. Then the loaded output is compared with the reference python object. So as far as I understand and intend, the tests are testing the right thing.

If you prefer, I can hardcode the json outputs and avoid using json.loads.

[ninsbl]

Nice. I wonder if it would make sense to create a standard parser option (e.g. G_OPT_M_FORMAT) for string output formats? Not needed for this PR, but could be helpful for wider implementation of JSON output across various modules...

[echoix]

Nice. I wonder if it would make sense to create a standard parser option (e.g. G_OPT_M_FORMAT) for string output formats? Not needed for this PR, but could be helpful for wider implementation of JSON output across various modules...

Totally agree that adding json output to many modules should be using a standard option, it will be duplicated many times otherwise.

[kritibirda26]

If someone can open an issue with more details on what to do, I'll be happy to implement it.

[echoix]

If someone can open an issue with more details on what to do, I'll be happy to implement it.

It's not quite finished discussing: #3410

[petrasovaa]

This looks great @kritibirda26, but now I wonder if we could try a different layout for the JSON. Currently the layout is similar to the csv output, but I was thinking something like:

{
  "data": [
    {
      "easting": 123.456,
      "northing": 78.901,
      "elevation": {"value": 345.67, "color": "#ff0000"},
      "slope": {"value": 12.3, "color": "#00ff00"},
      "aspect": {"value": 45.6, "color": "#0000ff"}
    },
    {
      "easting": 234.567,
      "northing": 89.012,
      "elevation": {"value": 456.78, "color": "#ff0000"},
      "slope": {"value": 23.4, "color": "#00ff00"},
      "aspect": {"value": 56.7, "color": "#0000ff"}
    }
  ]
}

or even simpler

[
    {
      "easting": 123.456,
      "northing": 78.901,
      "elevation": {"value": 345.67, "color": "#ff0000"},
      "slope": {"value": 12.3, "color": "#00ff00"},
      "aspect": {"value": 45.6, "color": "#0000ff"}
    },
    {
      "easting": 234.567,
      "northing": 89.012,
      "elevation": {"value": 456.78, "color": "#ff0000"},
      "slope": {"value": 23.4, "color": "#00ff00"},
      "aspect": {"value": 56.7, "color": "#0000ff"}
    }
]

Would it be sufficiently easy to redo it this way?

[petrasovaa]

If someone can open an issue with more details on what to do, I'll be happy to implement it.

Let's leave this for later.

At this point it would be good to draft a proposal based on the template specified here and introduce yourself on the grass-dev mailing list. See the tips on the idea page as well. Share the proposal in a google doc or similar so that we can comment. Let us know if you have questions!

[cwhite911]

This looks great @kritibirda26, but now I wonder if we could try a different layout for the JSON. Currently the layout is similar to the csv output, but I was thinking something like:

{
  "data": [
    {
      "easting": 123.456,
      "northing": 78.901,
      "elevation": {"value": 345.67, "color": "#ff0000"},
      "slope": {"value": 12.3, "color": "#00ff00"},
      "aspect": {"value": 45.6, "color": "#0000ff"}
    },
    {
      "easting": 234.567,
      "northing": 89.012,
      "elevation": {"value": 456.78, "color": "#ff0000"},
      "slope": {"value": 23.4, "color": "#00ff00"},
      "aspect": {"value": 56.7, "color": "#0000ff"}
    }
  ]
}

or even simpler

[
    {
      "easting": 123.456,
      "northing": 78.901,
      "elevation": {"value": 345.67, "color": "#ff0000"},
      "slope": {"value": 12.3, "color": "#00ff00"},
      "aspect": {"value": 45.6, "color": "#0000ff"}
    },
    {
      "easting": 234.567,
      "northing": 89.012,
      "elevation": {"value": 456.78, "color": "#ff0000"},
      "slope": {"value": 23.4, "color": "#00ff00"},
      "aspect": {"value": 56.7, "color": "#0000ff"}
    }
]

Would it be sufficiently easy to redo it this way?

It might be worth returning the data as geojson.

{
  "type": "FeatureCollection",
  "bbox": [100.0, 0.0, 105.0, 1.0],
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [123.456, 78.901]
      },
      "properties": {
        "elevation": {
          "value": 345.67,
          "color": "#ff0000"
        },
        "slope": {
          "value": 12.3,
          "color": "#00ff00"
        },
        "aspect": {
          "value": 45.6,
          "color": "#0000ff"
        }
      }
    },
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [234.567, 89.012]
      },
      "properties": {
        "elevation": {
          "value": 456.78,
          "color": "#ff0000"
        },
        "slope": {
          "value": 23.4,
          "color": "#00ff00"
        },
        "aspect": {
          "value": 56.7,
          "color": "#0000ff"
        }
      }
    }
  ]
}

[petrasovaa]

It might be worth returning the data as geojson.

Good point, but perhaps as an additional format? I think having a simple json covers most cases, you just want the value.

[kritibirda26]

This looks great @kritibirda26, but now I wonder if we could try a different layout for the JSON. Currently the layout is similar to the csv output, but I was thinking something like:
@petrasovaa, The actual json output doesn't have any elevation or aspect data so to confirm this is about changing items in the array from

{
    "easting": "332533.5941495",
    "northing": "242831.139883875",
    "site_name": "",
    "boundary_county_500m": 121,
    "boundary_county_500m_color": "010:000:255",
}

to

{
    "easting": "332533.5941495",
    "northing": "242831.139883875",
    "site_name": "",
    "boundary_county_500m": { "value": 121, "color": "010:000:255" }
}

[petrasovaa]

@petrasovaa, The actual json output doesn't have any elevation or aspect data so to confirm this is about changing items in the array from

Yes, elevation and aspect were just examples of raster layers.

[kritibirda26]

Hi! @cwhite911 and @petrasovaa, I have applied the suggestions and also updated the json format as suggested.

[petrasovaa]

@kritibirda26 since this is getting close to being done, would you like to start working on the GSoC proposal? Deadline is next week.

[kritibirda26]

@petrasovaa I have started working on the GSoC proposal. I aim to share a draft by tomorrow on the mailing list. I have one question about it. For a 350 hours project, how many tools should I aim to add JSON support for?

[petrasovaa]

@petrasovaa I have started working on the GSoC proposal. I aim to share a draft by tomorrow on the mailing list. I have one question about it. For a 350 hours project, how many tools should I aim to add JSON support for?

Good question, I will have to think about it, some tools will be more work than others. You can try to guess also based on how much time you spent on this PR. And it's more about the quality. Start the proposal and I and others will comment on it and prioritize the tools.

[cwhite911]

@petrasovaa second option (the one that is implemented here) integrates really well with pandas dataframes.

import pandas as pd

grass_out = gs.read_command("r.what", map="elevation,slope,aspect", points=samples, format="json", flags="r")
# example output
grass_out = [
    {
      "easting": 123.456,
      "northing": 78.901,
      "elevation": {"value": 345.67, "color": "#ff0000"},
      "slope": {"value": 12.3, "color": "#00ff00"},
      "aspect": {"value": 45.6, "color": "#0000ff"}
    },
    {
      "easting": 234.567,
      "northing": 89.012,
      "elevation": {"value": 456.78, "color": "#ff0000"},
      "slope": {"value": 23.4, "color": "#00ff00"},
      "aspect": {"value": 56.7, "color": "#0000ff"}
    }
]

df_n = pd.json_normalize(json.loads(grass_out), max_level=1)
df_n.head()

Outputs

	easting	northing	elevation.value	elevation.color	slope.value	slope.color	aspect.value	aspect.color
0	123.456	78.901	345.67	#ff0000	12.3	#00ff00	45.6	#0000ff
1	234.567	89.012	456.78	#ff0000	23.4	#00ff00	56.7	#0000ff

[petrasovaa]

This looks ready to go!