fix(bq,sf,rs,pg|quadbin): improve precision of long lat conversion near the latitude limits

[jgoizueta]

Description

Shortcut

• Story: https://app.shortcut.com/cartoteam/story/373977/improve-precision-of-long-lat-conversion-near-the-latitude-limits
• Autolink: [sc-373977]

Experimental improvement of the QUADBIN_FROMLONGLAT lattitude clipping.

The formula that computes the quadbin Y coordinate from the latitude is unstable near the latitude limits (+/- 85.05112877980659). Because of this, we were clipping the latitude with a smaller value (85.05) to avoid errors that would assign an incorrect quadbin value to latitudes near the limit.

It seems better to clip the values of LN((1 + __sinlat) / (1 - __sinlat)) / pi instead of the input latitude.

A clipping of the input is maintained to avoid errors with latitude=90 or incorrectvalues with latitude>90 (same for negative values). Note it's questionable to allow and clip values greater than 90, because BigQuery would not allow creating a geography with them, and in any case they could be interpreted modulo 90 (so that 130 could be interpreted as 40, and not clipped).

Acceptance

SELECT `carto-un`.carto.QUADBIN_TOZXY(`carto-un`.carto.QUADBIN_FROMLONGLAT(0, 85.051128779806, 26))
[{
  "f0_": {
    "z": "26",
    "x": "33554432",
    "y": "2438"
  }
}]

SELECT `carto-un`.carto.QUADBIN_TOZXY(`cartodb-data-engineering-team`.jgoizueta_carto.QUADBIN_FROMLONGLAT(0, 85.051128779806, 26))
[{
  "f0_": {
    "z": "26",
    "x": "33554432",
    "y": "0"
  }
}]

SELECT `carto-un`.carto.QUADBIN_TOZXY(`carto-un`.carto.QUADBIN_FROMLONGLAT(0, 85.051128779806, 20))
[{
  "f0_": {
    "z": "20",
    "x": "524288",
    "y": "38"
  }
}]

SELECT `carto-un`.carto.QUADBIN_TOZXY(`cartodb-data-engineering-team`.jgoizueta_carto.QUADBIN_FROMLONGLAT(0, 85.051128779806, 20))
[{
  "f0_": {
    "z": "20",
    "x": "524288",
    "y": "0"
  }
}]

For negative latitudes:

SELECT `carto-un`.carto.QUADBIN_TOZXY(`carto-un`.carto.QUADBIN_FROMLONGLAT(0, -85.051128779806, 26))
[{
  "f0_": {
    "z": "26",
    "x": "33554432",
    "y": "67106425"
  }
}]

SELECT `carto-un`.carto.QUADBIN_TOZXY(`cartodb-data-engineering-team`.jgoizueta_carto.QUADBIN_FROMLONGLAT(0, -85.051128779806, 26))
[{
  "f0_": {
    "z": "26",
    "x": "33554432",
    "y": "67108863"
  }
}]

SELECT `carto-un`.carto.QUADBIN_TOZXY(`carto-un`.carto.QUADBIN_FROMLONGLAT(0, -85.051128779806, 20))
[{
  "f0_": {
    "z": "20",
    "x": "524288",
    "y": "1048537"
  }
}]

SELECT `carto-un`.carto.QUADBIN_TOZXY(`cartodb-data-engineering-team`.jgoizueta_carto.QUADBIN_FROMLONGLAT(0, -85.051128779806, 20))
[{
  "f0_": {
    "z": "20",
    "x": "524288",
    "y": "1048575"
  }
}]

[vdelacruzb]

Once this is merged: CartoDB/quadbin-py#17
We will have to update the version of the Redshift module.

[shortcut-integration]

This pull request has been linked to Shortcut Story #373977: Improve precision of long lat conversion near the latitude limits.

[github-actions]

Dedicated carto environment deployed in Redshift host redshift-serverless-shared.515242991920.us-east-1.redshift-serverless.amazonaws.com and carto_dev_data database

[github-actions]

Dedicated environment removed