Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up interpolation during delay calculation #98

Merged
merged 2 commits into from
Aug 1, 2020

Conversation

Askaholic
Copy link
Collaborator

@Askaholic Askaholic commented Jul 31, 2020

I profiled the zenith delay calculation at query points using the following command:

raiderDelay.py --date 20200103 --time 23:00:00 --latlon lat.dat lon.dat --model HRRR --zref 20000 -v --out test_output/

Where my lat/lon file consisted of 100x100 points spaced equally over the bounding box (SNWE) 37 40 -120 -116.

I found that the time was pretty evenly split between interpolation and reprojection:

         963425 function calls (913599 primitive calls) in 20.541 seconds

   Ordered by: internal time
   List reduced from 2679 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2    8.496    4.248    8.496    4.248 {method '_transform' of 'pyproj._transformer._Transformer' objects}
        2    5.090    2.545    6.403    3.202 interpolate.py:2499(_evaluate_linear)
        2    1.971    0.985    2.926    1.463 interpolate.py:2519(_find_indices)
60259/20139    1.278    0.000    2.736    0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}
        6    0.954    0.159    0.954    0.159 {method 'searchsorted' of 'numpy.ndarray' objects}
        2    0.432    0.216    0.432    0.216 {built-in method scipy.interpolate.interpnd._ndim_coords_from_arrays}
       40    0.164    0.004    0.164    0.004 {method 'astype' of 'numpy.ndarray' objects}
        2    0.148    0.074    0.148    0.074 interpolate.py:2505(<listcomp>)
        1    0.145    0.145    0.145    0.145 {RAiDER.makePoints.makePoints1D}
        2    0.140    0.070    0.140    0.070 {pyproj._transformer.from_crs}

Where the cumulative time spent in interpolation made up about 50% of the total computation time:

  ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.049    0.025    9.811    4.906 interpolate.py:2450(__call__)

After replacing this with the C++ interpolator, the interpolation which took 10 seconds before, took less than 1 second:

         963731 function calls (913898 primitive calls) in 11.178 seconds

   Random listing order was used
   List reduced from 2655 to 13 due to restriction <'interpolate'>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.001    0.001   10.652   10.652 delay.py:28(interpolateDelay)
        2    0.052    0.026    0.769    0.385 delayFcns.py:303(interpolate2)
        2    0.246    0.123    0.246    0.123 {built-in method RAiDER.interpolate.interpolate}

I also verified that the results are the same:

>>> old = gdal_open("test_output/old/HRRR_hydro_23_00_00_std.envi")
>>> new = gdal_open("test_output/HRRR_hydro_23_00_00_std.envi")
>>> np.nanmean((old-new)**2)
0.0

Copy link
Owner

@dbekaert dbekaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Askaholic this looks good to me. @jlmaurer any issues to merge?

Copy link
Collaborator

@jlmaurer jlmaurer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@jlmaurer jlmaurer merged commit 937ca73 into dbekaert:dev Aug 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants