Proj 6.1.1 to 8.0.1 performance issues and regressions

[rconde01]

Example of problem

#include <filesystem>
#include <fstream>
#include <iostream>
#include <stdexcept>
#include <string>
#include <vector>

#include "gdal_priv.h"

enum class EPSGCodes {
    // https://epsg.io/4326
    WGS84_Horizontal = 4326,

    // https://epsg.io/4979
    WGS84_Vertical = 4979,

    // https://epsg.io/5773
    EGM96_Vertical = 5773
};

// Hold input data
struct data {
    std::vector<double> latitudes;
    std::vector<double> longitudes;
    std::vector<double> elevations;
};

// batches of data
std::vector<data> data_groups;

// Load the data we want to transform
void load_data(std::filesystem::path const & exe_dir){
    auto data_path = (exe_dir / "terrain_data.dat").string();

    std::ifstream f(data_path,std::ios::binary);

    if(!f)
        throw std::runtime_error("");

    while(!f.eof()){
        int count{};

        f.read(reinterpret_cast<char *>(&count),sizeof(count));

        data_groups.push_back(data{});

        auto & lat = data_groups.back().latitudes;
        auto & lon = data_groups.back().longitudes;
        auto & elv = data_groups.back().elevations;

        lat.reserve(count);
        lon.reserve(count);
        elv.reserve(count);

        for(int i = 0; i < count; ++i){
            double v;

            f.read(reinterpret_cast<char *>(&v),sizeof(double));
            lat.push_back(v);

            f.read(reinterpret_cast<char *>(&v),sizeof(double));
            lon.push_back(v);

            f.read(reinterpret_cast<char *>(&v),sizeof(double));
            elv.push_back(v);
        }
    }
}

// transform the data
void transform_data(std::filesystem::path const & exe_dir){
    std::string proj_data_path = (exe_dir / "./proj").string();

    const char* proj_search_paths[] = {proj_data_path.c_str(), nullptr};

    OSRSetPROJSearchPaths(proj_search_paths);

    OGRSpatialReference source_horizontal;

    if(source_horizontal.importFromEPSG(
        static_cast<int>(EPSGCodes::WGS84_Horizontal)) != OGRERR_NONE)
        throw std::runtime_error("");

    OGRSpatialReference source_vertical;

    if(source_vertical.importFromEPSG(
        static_cast<int>(EPSGCodes::EGM96_Vertical)) != OGRERR_NONE)
        throw std::runtime_error("");

    OGRSpatialReference source_coordinate_system;

    if(source_coordinate_system.SetCompoundCS(
        "WGS84 Horizontal + EGM96 Vertical",
        &source_horizontal,
        &source_vertical) != OGRERR_NONE)
        throw std::runtime_error("");

    OGRSpatialReference dest_coordinate_system;

    if(dest_coordinate_system.importFromEPSG(
        static_cast<int>(EPSGCodes::WGS84_Vertical)) != OGRERR_NONE)
        throw std::runtime_error("");

    auto transform = 
        OGRCreateCoordinateTransformation(
            &source_coordinate_system,
            &dest_coordinate_system);

    for(auto & d : data_groups){
        transform->Transform(
            static_cast<int>(d.elevations.size()),
            d.latitudes.data(),
            d.longitudes.data(),
            d.elevations.data(),
            nullptr);
    }
}

int main(int argc, char ** argv){
    std::filesystem::path exe_path(argv[0]);
    auto exe_dir = exe_path.parent_path();

    std::cout << "Load Data" << std::endl;
    load_data(exe_dir);

    std::cout << "Transform Data" << std::endl;
    transform_data(exe_dir);
}

Problem description

I have compiled and run the above example on visual studio 2019 with a preprocessed data file (~500 mb of lat/lon/elevation data). I run it against gdal/proj 3.0.1/6.1.1 and 3.2.1/8.0.1.

• 3.0.1/6.1.1 takes 17.9 seconds to run
• 3.2.1/8.0.1 takes 21.6 seconds to run
• Loading the data takes ~3.3 seconds in both cases
• So the newer version is ~25% slower

The vTune profile from 6.1.1 is:

The vTune profile from 8.0.1 is:

Besides the performance regression, it is notable that about 25% of the runtime in both cases is spent just calling proj_errno_reset...not sure if anything can be done about that.

Environment Information

• PROJ version (proj) : 6.1.1/8.0.1
• Operation System Information: Windows 10

Installation method

• from source

[rouault]

Are the longitude, latitude covering the whole world/a large extent or a small extent ? And if covering a large extent, are consecutive points in "random order" or are they generally grouped by location ? I'm asking this since the us_nga_egm96_15.tif uses 256x256 tiles (18 tiles for the whole world), and PROJ has a cache for 12 tiles per file. Could you also expand the vTune profile for GTiffVGrid::valueAt ?

[rconde01]

Here's a summary of the point-count, south-west, and north-east bounds for each data group:
terrain_regions.txt

It's a small region near 37 latitude, -119 longitude. It should be scanning west to east, south to north.

Here's the expanded detail in the profile: