Validation against AEIC v2

[ian-ross]

@aditeyashukla has already begun work on this, but I wanted to start a discussion to pin down exactly what we want from this exercise.

From the discussion we had at the last meeting, I think we have the following requirements:

1. Select a validation set of 20-30 missions. (Ideally randomly, though we want to impose some conditions on the cases that we use in terms of coverage of different conditions, so we might have to adjust the selection.)
2. Avoid pathological cases for the main validation tests, but check on a few separately. ("Pathological cases" in this sense mean short missions, less than 200 km, which cause a problem for the legacy trajectory builder, and maximum range missions.)
3. The set of cases should cover a representative range of different conditions: different numbers of engines, different aircraft types, different distances, and should include missions that cross the dateline and that run over UTC and/or local midnight.
4. We will want to use fake performance data, so that we can run a subset of the validation cases as automated tests without needing to worry about protecting proprietary aircraft performance information.
5. Input data for the validation process (performance models, mission definitions, etc.) as well as output from the AEIC v2 Matlab code should be provided as a single ZIP file, and the data generation process should be documented. (The code for this process may have to be in a separate private repository to avoid proprietary data concerns.)
6. Running the validation cases and comparing results for the new Python AEIC should be fully automated. Some validation cases should be run in normal tests, but there should be a single command to run the full validation set.

I think this leads to the following list of tasks (with provisional assignments of people to do them):

• Initial selection of test cases for automation, plus some pathological cases for separate review (@aditeyashukla);
• Generation of fake performance data (@aditeyashukla);
• Review and agreement of test cases and fake performance models (everyone);
• Running test cases in AEIC v2 (@aditeyashukla);
• Generation of input and comparison data set (@aditeyashukla and/or @ian-ross);
• Validation script for new AEIC (@ian-ross and/or @aditeyashukla);
• Diagnose and correct differences between old AEIC and new AEIC (@ian-ross and/or @aditeyashukla);
• Add subset of validation cases to CI (@ian-ross).

Is there anything missing from this plan? Does anyone have any comments they'd like to make in general about this process? And does anyone have any ideas on calibration of expectations for this process in general?

The initial results that Adi showed for the trajectory output were promising, and it doesn't seem unreasonable to expect to be able to get to result identical to the old AEIC, but how close do we really need to get? If we're off by ±1%, is that OK? ±5%?

[ian-ross]

First steps on a validation script: #85

[ian-ross]

We're going to assume that things are good if we can complete the validation against the small mission set that we've started with, since we're going to have the 2019 gridded annual emissions to check as well.