Writing the evaluation spec for simplified data collection

[xubowenhaoren]

Hello,

As we discussed in a related issue, I am working on a project with data collection and analysis. However, due to time & resource constraints, we have to limit the data collection to:

• One phone with no control group
• One round of data collection, meaning each trip is only performed once

However, this simplified data collection still meets the purposes of this data collection work, which do not focus on the device power consumption but on the model prediction accuracy:

1. Collect high-accuracy data from both e-mission (MobilityNet) and an IMU sensor
2. Train a sample model seed based on the ground truth of the pre-defined trips
3. Document the procedure to allow a continuous improvement of the model seed, provided that we have more pre-defined trips and ground truth

Thus I wish to know how to write the evaluation spec to define our simplified data collection.

• phones: Should I just include one evaluation phone?
• calibration_tests: If I choose to only use high-accuracy high-frequency sensing, what sets of tests should I keep?
• sensing_settings: same question to the calibration_tests.
• setup_notes: Can I simply leave it empty?

[xubowenhaoren]

Another question regarding the actual evaluation trip: When the natural trip ends, the e-mission app may not immediately stop the motion sensing and/or trip detecting. Can I use the Profile -> developer options and force the state to "end trip" to save time? Or must I wait for the e-mission processing to finish?

[shankari]

@xubowenhaoren responses inline

phones: Should I just include one evaluation phone?

Yes

calibration_tests: If I choose to only use high-accuracy high-frequency sensing, what sets of tests should I keep?

You can skip calibration, which is designed for use with multiple phones, to ensure that the phones behave in the same way if they have identical settings

sensing_settings: same question to the calibration_tests.

I assume you would keep the accuracy control since you don't care about the power and you do care about the accuracy.

setup_notes: Can I simply leave it empty?

Sure

[shankari]

Another question regarding the actual evaluation trip: When the natural trip ends, the e-mission app may not immediately stop the motion sensing and/or trip detecting. Can I use the Profile -> developer options and force the state to "end trip" to save time? Or must I wait for the e-mission processing to finish?

If you use the accuracy control, e-mission will never stop the motion sensing and/or trip detecting. This allows you to come up with trip segmentation algorithms separate of what e-mission does in the app. the evaluation UI will allow you to specify when the trip ends.

I think I force synced a couple of times just before the end of the evaluation to be on the safe side.

[xubowenhaoren]

Another question regarding the actual evaluation trip: When the natural trip ends, the e-mission app may not immediately stop the motion sensing and/or trip detecting. Can I use the Profile -> developer options and force the state to "end trip" to save time? Or must I wait for the e-mission processing to finish?

If you use the accuracy control, e-mission will never stop the motion sensing and/or trip detecting. This allows you to come up with trip segmentation algorithms separate of what e-mission does in the app. the evaluation UI will allow you to specify when the trip ends.

I think I force synced a couple of times just before the end of the evaluation to be on the safe side.

I see. By "force sync", do you refer to this button?

[xubowenhaoren]

In Option 2a/2b of the spec_creation/create_ground_truth_for_legs.ipynb, there was the following comment specifying the motion modes:

"mode": "WALKING", # or CAR, BICYCLING or BUS

I realize that this list is much shorter than the motion modes available in e-mission, such as:

      "WALKING": 'brown',
      "RUNNING": 'brown',
      "BICYCLING": 'green',
      "IN_VEHICLE": 'purple',
      "LIGHT_RAIL": 'blue',
      "BUS": 'navy',
      "TRAIN": 'skyblue',
      "TRAM": 'slateblue',
      "SUBWAY": 'darkcyan',
      "CAR": 'salmon',
      "UNKNOWN": 'orange',
      "UNPROCESSED": 'orange',
      "AIR_OR_HSR": "red"

In our current design of the pre-defined trips, we plan to include transportation via train/subway. So what do we need to specify for the motion modes in spec_creation/create_ground_truth_for_legs.ipynb?

[shankari]

I see. By "force sync", do you refer to this button?

Yes.

In our current design of the pre-defined trips, we plan to include transportation via train/subway. So what do we need to specify for the motion modes in spec_creation/create_ground_truth_for_legs.ipynb?

OSRM doesn't support transit trips since they require route timing information; it works purely on the OSM network. You would have to use the OSM relationid option (Option 1) instead.

[xubowenhaoren]

What I actually like about Option 2b of the spec_creation/create_ground_truth_for_legs.ipynb is that I can define trip routes using the latitudes and longitudes. I've even outlined a procedure for converting Google Maps routes to a list of lat/longs. I wish to continue using Google Maps to both plan for the pre-defined trips and draw the equivalent route.

OSRM doesn't support transit trips since they require route timing information.

• When I click "start/end/next/stop" in the evaluation page, you mean it doesn't log the start/end times and thus the temporal truth?
• Should I only pre-define trips with CAR, BICYCLING, BUS, WALKING then?

[shankari]

I wish to continue using Google Maps to both plan for the pre-defined trips and draw the equivalent route.

I believe Google Maps (if you are using the API) also returns polylines directly. You can use the polyline directly in the sample spec.

Alternatively, you can encode lat/lon points directly as a polyline - there are examples of polyline encoding as well.

You should make sure that the lat/lon points are dense, though, specially at the curves in the roads so we have good ground truth for trajectory matching. Not sure how dense KML values are.

When I click "start/end/next/stop" in the evaluation page, you mean it doesn't log the start/end times and thus the temporal truth?

OSRM is one of many ways to create the ground truth spec. The relation id is another. Google Maps is another. Once you have created the spec, there is no indication where it came from, and all specs work the same way irrespective of the modes. Please see the other specs in final_sfbayarea for examples of other multi-modal specs.

[xubowenhaoren]

In the discussion today, I realized that for future researchers interested in using Google Maps to generate both the polyline route and have the navigation link available for a trip, you may directly use my notebook. See the example here to encode the lat/long list to polyline.

import polyline
polyline.encode([(38.5, -120.2), (40.7, -120.9), (43.2, -126.4)], 5)

[shankari]

For the record, the emevalzephyr skin (with my fixes) will show you the next step (with trajectory) along your route. It does not have real-time navigation, but although one of the requests is to add a "blue dot" indicating where you are.