Pressure does not decrease monotonically in your sounding.

[ArtimisFowl888]

When I try to find the Parcel_profile with data from 72357 OUN Norman Observations at 00Z 31 May 2013 I get the following error.

metpy.calc.exceptions.InvalidSoundingError: Pressure does not decrease monotonically in your sounding. Using scipy.signal.medfilt may fix this.

The issue is that the data fails in thermo.py at:

def _check_pressure(pressure): """Check that pressure decreases monotonically. Returns True if the pressure decreases monotonically; otherwise, returns False. """ return np.all(pressure[:-1] > pressure[1:])

Should np.all(pressure[:-1] > pressure[1:]) be np.all(pressure[:-1] >= pressure[1:]) because in the event that pressure[:-1] = pressure[1:] the function fill retun false.

[mvoss-wx]

For what it's worth...I get the same issue "sometimes" when using metpy 0.10.2.

The origin in the latest case is plotting a parcel profile and not the sounding itself:
prof = mpcalc.parcel_profile(p, T[0], Td[0]).to('degC')

When I plot the same parcel profile using metpy 1.0.1 the error does not occur, in the current case at least. Not sure how general this is and I seem to recall getting this error for perfectly good soundings and not understanding why.

[mvoss-wx]

One more comment. I confirmed this breaks again in metpy 1.1.0

so to summarize for me:
breaks in 0.10.2
works in 1.0.1
breaks in 1.1.0

[sgdecker]

As the error indicates, the data you are using doesn't contain pressures that strictly decrease with height. However, I think it is a bug that parcel_profile returns an invalid sounding from the perspective of this code. It may be that the restriction should be relaxed to a requirement that the pressure not increase (so that repeating the same pressure twice is OK), but I'm not sure if that is a "safe" thing to do in the sense that it might cause some MetPy function to return bogus results in certain instances.

It is definitely the case that real data does repeat pressures. For instance, with this code:

from datetime import datetime
from siphon.simplewebservice.wyoming import WyomingUpperAir

sounding_time = datetime(2013, 5, 31, 0)
station = 'OUN'
sounding_df = WyomingUpperAir.request_data(sounding_time, station)

print(sounding_df.tail(20))

we get

     pressure  height  temperature  ...  longitude  elevation     pw
125      14.8   28651        -48.4  ...     -97.44      345.0  33.39
126      13.4   29318        -45.5  ...     -97.44      345.0  33.39
127      12.9   29566        -45.4  ...     -97.44      345.0  33.39
128      11.8   30175        -45.2  ...     -97.44      345.0  33.39
129      11.8   30165        -45.3  ...     -97.44      345.0  33.39
130      11.3   30480        -42.4  ...     -97.44      345.0  33.39
131      10.8   30785        -39.5  ...     -97.44      345.0  33.39
132      10.7   30827        -39.1  ...     -97.44      345.0  33.39
133      10.3   31090        -39.6  ...     -97.44      345.0  33.39
134      10.1   31222        -39.9  ...     -97.44      345.0  33.39
135      10.0   31290        -39.5  ...     -97.44      345.0  33.39
136       9.8   31394        -38.7  ...     -97.44      345.0  33.39
137       9.5   31644        -36.7  ...     -97.44      345.0  33.39
138       9.4   31699        -36.6  ...     -97.44      345.0  33.39
139       8.6   32309        -35.6  ...     -97.44      345.0  33.39
140       7.9   32918        -34.7  ...     -97.44      345.0  33.39
141       7.3   33528        -33.7  ...     -97.44      345.0  33.39
142       7.0   33833        -33.4  ...     -97.44      345.0  33.39
143       7.0   33780        -33.3  ...     -97.44      345.0  33.39
144       6.9   33881        -33.5  ...     -97.44      345.0  33.39

[20 rows x 15 columns]

where we see both 11.8 mb and 7.0 mb are repeated (presumably due to rounding) in the Norman sounding referenced above.

[sgdecker]

Interestingly, when looking at this data in GEMPAK (downloaded from the ISU GEMPAK archive), I see:

1. GEMPAK has the pressure levels to the hundredth of a millibar, reducing the risk of duplication (and there is code in the decoder routine that I think would prevent a duplicated pressure anyway).
2. Tons of discrepancies on the non-mandatory levels in the stratosphere (pressure less than 100 mb). Here's what SNLIST has toward the top of the sounding (pressure and temperature):

     14.86   -43.84
     12.95   -42.34
     11.82   -41.33
     11.29   -40.83
     10.79   -40.33
     10.30   -39.83
     10.00   -39.50
      9.85   -39.24
      9.43   -38.48
      8.64   -36.96
      7.92   -35.45
      7.26   -33.93
      7.00   -33.30

For example, at 9.4 (or 9.43) mb, the data returned via Siphon has a temperature of -36.6 C, but GEMPAK has -38.48 C. I guess we'd have to dredge up the original text bulletin to know which is right.

[sgdecker]

Finally, using the ISU reader shows NaN for the temperature at each of the non-mandatory levels where GEMPAK has a temperature (this might be the difference between a "merged" and "unmerged" sounding?):

from datetime import datetime
from siphon.simplewebservice.iastate import IAStateUpperAir

sounding_time = datetime(2013, 5, 31, 0)
station = 'OUN'
sounding_df = IAStateUpperAir.request_data(sounding_time, station)

print(sounding_df.tail(20))

gives:

     pressure   height  temperature  ...       time     u_wind     v_wind
123      15.2  28482.0        -49.1  ... 2013-05-31        NaN        NaN
124      14.8  28651.0          NaN  ... 2013-05-31 -14.772116   2.604723
125      13.4  29313.0        -45.5  ... 2013-05-31        NaN        NaN
126      12.9  29565.0          NaN  ... 2013-05-31 -19.318517  -5.176381
127      11.8  30159.0        -45.3  ... 2013-05-31        NaN        NaN
128      11.7  30176.0          NaN  ... 2013-05-31 -11.817693  -2.083778
129      11.3  30480.0          NaN  ... 2013-05-31 -11.313708 -11.313708
130      10.8  30784.0          NaN  ... 2013-05-31 -16.914467  -6.156363
131      10.7  30819.0        -39.1  ... 2013-05-31        NaN        NaN
132      10.3  31089.0          NaN  ... 2013-05-31 -19.318517   5.176381
133      10.1  31213.0        -39.9  ... 2013-05-31        NaN        NaN
134      10.0  31290.0        -39.5  ... 2013-05-31 -17.202193  12.045105
135       9.8  31394.0          NaN  ... 2013-05-31 -10.324376  14.744737
136       9.5  31642.0        -36.7  ... 2013-05-31        NaN        NaN
137       9.4  31699.0          NaN  ... 2013-05-31  -9.192533   7.713451
138       8.6  32308.0          NaN  ... 2013-05-31 -14.488887  -3.882286
139       7.9  32918.0          NaN  ... 2013-05-31 -38.407502  -6.772279
140       7.3  33528.0          NaN  ... 2013-05-31 -32.498656  -5.730390
141       7.0  33780.0        -33.3  ... 2013-05-31 -24.432008  -8.892524
142       6.9  33881.0        -33.5  ... 2013-05-31        NaN        NaN

[20 rows x 10 columns]

The ISU reader doesn't return a monotonic sounding either, but in this case the repeated pressures are 71.6 mb and 55.8 mb.

[sgdecker]

Looking at these results, I see four possible ways to address the fundamental issue:

1. Add options to the relevant siphon functions that force those functions to return a monotonic sounding.
2. Add a sanitize_sounding function to MetPy that eliminates duplicate pressures from soundings in some standard way.
3. Relax the monotonicity condition to a non-increasing condition, and then add checks in the various thermodynamic calculations where having a repeated pressure may produce an erroneous result. If all such cases throw exceptions, this would be relatively straightforward to do (add try/except blocks), and this increases MetPy's flexibility of the code (who cares about bad data in the stratosphere when you are only plotting/calculating things in the troposphere), but this would be dangerous if:

• There are instances where this change doesn't throw an exception, but an incorrect result is calculated anyway, and
• Every possible corner case of this type is not already covered by the test suite.
  In other words, option 3 would eliminate the monotonicity crash for cases like this Norman sounding where pressures are repeated, but may happily produce an incorrect result w/o warning in some other cases unless we go through all of the thermodynamic sounding code very carefully.

4. Do nothing, and the onus is on the user to verify that siphon (or whatever the data source is) is returning data that is monotonic, and if not, take corrective action prior to calling MetPy sounding functions.

[dopplershift]

For example, at 9.4 (or 9.43) mb, the data returned via Siphon has a temperature of -36.6 C, but GEMPAK has -38.48 C. I guess we'd have to dredge up the original text bulletin to know which is right.

So Siphon is correctly returning what Wyoming gives us:

    9.5  31644  -36.7  -53.7     16   2.71    133     13  894.4  925.7  895.9
    9.4  31699  -36.6  -53.6     16   2.76    130     12  896.8  928.7  898.3
    8.6  32309  -35.6  -52.6     16   3.39     75     15  923.1  963.4  925.0

Not sure how they're doing their stuff. It never occurred to me before that what we're really suffering from here is essentially that Wyoming isn't giving us all the digits available. 😞

When we put in the check, we thought it was only rare, errant data that would trigger it--not regular, normal data due to data access issues. The check is only currently used by parcel_profile and parcel_profile_with_lcl, and of course anything that calls those functions. I can say that changing _check_pressure to ensure non-increasing doesn't break any current tests.

I'll also add that the purpose of that check was to give better errors than random errors out of the internal scipy/numpy calculations--not as a strict correctness check. So I'm inclined to make the change from > to >= and wait until we find a case where that's a problem.

[sgdecker]

That's definitely the easiest way to fix this!