Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

postgresqlPackages.postgis: build minimal gdal version #276348

Merged
merged 1 commit into from Jan 19, 2024
Merged

postgresqlPackages.postgis: build minimal gdal version #276348

merged 1 commit into from Jan 19, 2024

Conversation

imincik
Copy link
Contributor

@imincik imincik commented Dec 23, 2023 •
edited

Description of changes

Build Postgis with minimal gdal version (gdalMinimal).

gdalMinimal variant is disabling support for following features :

  • useTiledb - TileDB
  • useLibHEIF - HEIF and AVIF (AV1 Image File Format) file format decoder and encoder
  • useLibJXL - JPEG XL (encoder and decoder)
  • useMysql - MySQL
  • usePostgres - PostgreSQL
  • usePoppler - PDF rendering library
  • useArrow - in-memory data
  • useHDF - The Hierarchical Data Format version (HDF)
  • useNetCDF - Unidata network Common Data Format (CDF)
  • useArmadillo - Armadillo is a high quality linear algebra library

Following is the difference in output of

SET postgis.gdal_enabled_drivers = 'ENABLE_ALL';

SELECT short_name, long_name, can_write
FROM st_gdaldrivers()
ORDER BY short_name

between PostGIS built with gdal and gdalMinimal:

--- pg_drivers-full.txt	2024-01-03 11:27:45.906789288 +0100
+++ pg_drivers-minimal.txt	2024-01-03 11:10:26.521792334 +0100
@@ -6,7 +6,6 @@
  AIG             | Arc/Info Binary Grid
  AirSAR          | AirSAR Polarimetric Image
  ARG             | Azavea Raster Grid format
- BAG             | Bathymetry Attributed Grid
  BIGGIF          | Graphics Interchange Format (.gif)
  BLX             | Magellan topo (.blx)
  BMP             | MS Windows Device Independent Bitmap
@@ -54,11 +53,6 @@
  GTiff           | GeoTIFF
  GTX             | NOAA Vertical Datum .GTX
  GXF             | GeoSoft Grid Exchange Format
- HDF4            | Hierarchical Data Format Release 4
- HDF4Image       | HDF4 Dataset
- HDF5            | Hierarchical Data Format Release 5
- HDF5Image       | HDF5 Dataset
- HEIF            | ISO/IEC 23008-12:2017 High Efficiency Image File Format
  HF2             | HF2/HFZ heightfield raster
  HFA             | Erdas Imagine Images (.img)
  HTTP            | HTTP Fetching Wrapper
@@ -72,7 +66,6 @@
  JDEM            | Japanese DEM (.mem)
  JP2OpenJPEG     | JPEG-2000 driver based on OpenJPEG library
  JPEG            | JPEG JFIF
- JPEGXL          | JPEG-XL
  KMLSUPEROVERLAY | Kml Super Overlay
  KRO             | KOLOR Raw
  L1B             | NOAA Polar Orbiter Level 1b Data Set
@@ -88,7 +81,6 @@
  MRF             | Meta Raster Format
  MSGN            | EUMETSAT Archive native (.nat)
  NDF             | NLAPS Data Format
- netCDF          | Network Common Data Format
  NGSGEOID        | NOAA NGS Geoid Height Grids
  NGW             | NextGIS Web
  NITF            | National Imagery Transmission Format
@@ -110,7 +102,6 @@
  PLSCENES        | Planet Labs Scenes API
  PNG             | Portable Network Graphics
  PNM             | Portable Pixmap Format (netpbm)
- PostGISRaster   | PostGIS Raster driver
  PRF             | Racurs PHOTOMOD PRF
  R               | R Object Data Store
  Rasterlite      | Rasterlite
@@ -121,7 +112,6 @@
  RRASTER         | R Raster
  RS2             | RadarSat 2 XML Product
  RST             | Idrisi Raster A.1
- S102            | S-102 Bathymetric Surface Product
  SAFE            | Sentinel-1 SAR SAFE Product
  SAGA            | SAGA GIS Binary Grid (.sdat, .sg-grd-z)
  SAR_CEOS        | CEOS SAR Image
@@ -137,7 +127,6 @@
  Terragen        | Terragen heightfield
  TGA             | TGA/TARGA Image File Format
  TIL             | EarthWatch .TIL
- TileDB          | TileDB
  TSX             | TerraSAR-X Product
  USGSDEM         | USGS Optional ASCII DEM (and CDED)
  VICAR           | MIPL VICAR file
@@ -150,5 +139,5 @@
  XYZ             | ASCII Gridded XYZ
  Zarr            | Zarr
  ZMap            | ZMap Plus Grid
-(150 rows)
+(139 rows)

This change is reducing closure size from 1.5G to
544M.

  • Closure size with full gdal
nix path-info -rsSh /nix/store/0l4kxyvzclrb0nvaf5aqw6q1rcmfbg6h-postgis-3.4.1

...

/nix/store/9bji3x8w4j2j679k9z8mm359nv5ivv84-gdal-3.8.2                     	  42.3M	   1.3G
/nix/store/0l4kxyvzclrb0nvaf5aqw6q1rcmfbg6h-postgis-3.4.1                  	  42.4M	   1.5G
  • Closure size with gdalMinimal
nix path-info -rsSh /nix/store/dghk2lkjbqz0k2i54lfw0c5hspgj8j3m-postgis-3.4.1
...

/nix/store/j1ybly3xvcqac0yngq7r37rvqp3p1amk-gdal-3.8.2                     	  38.5M	 308.1M
/nix/store/dghk2lkjbqz0k2i54lfw0c5hspgj8j3m-postgis-3.4.1                  	  42.4M	 544.1M

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 馃憤 reaction to pull requests you find important.

@imincik imincik requested review from marsam and removed request for marsam December 23, 2023 17:56
@imincik imincik self-assigned this Dec 23, 2023
@das-g
Copy link
Member

das-g commented Dec 23, 2023 •
edited

Will this limit the features of PostGIS or does PostGIS only ever use GDAL/OGR features also provided by gdalMinimal?

@imincik
Copy link
Contributor Author

imincik commented Jan 3, 2024 •
edited

Will this limit the features of PostGIS or does PostGIS only ever use GDAL/OGR features also provided by gdalMinimal?

Sorry for late answer. I updated PR description with the list of disabled features in gdalMinimal.

GDAL should be used only for Postgis Raster extension.

das-g
das-g approved these changes Jan 3, 2024
View reviewed changes
Copy link
Member

@das-g das-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable.

@imincik imincik requested a review from das-g January 3, 2024 10:34
@imincik
postgresqlPackages.postgis: build minimal gdal version
afc2f0f 
This change is reducing closure size from 1.5G to
544M.
@imincik
Copy link
Contributor Author

imincik commented Jan 3, 2024

@ofborg build postgresqlPackages.postgis

@imincik
Copy link
Contributor Author

imincik commented Jan 3, 2024

@marsam , are you OK with this change ?

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/week-in-geospatial-team/37035/2

@nialov
Copy link
Contributor

nialov commented Jan 14, 2024

Result of nixpkgs-review pr 276348 run on x86_64-linux 1

20 packages built:
  • postgresql12JitPackages.postgis
  • postgresql12JitPackages.postgis.doc
  • postgresql12Packages.postgis
  • postgresql12Packages.postgis.doc
  • postgresql13JitPackages.postgis
  • postgresql13JitPackages.postgis.doc
  • postgresql13Packages.postgis
  • postgresql13Packages.postgis.doc
  • postgresql14JitPackages.postgis
  • postgresql14JitPackages.postgis.doc
  • postgresql14Packages.postgis
  • postgresql14Packages.postgis.doc
  • postgresqlJitPackages.postgis (postgresql15JitPackages.postgis)
  • postgresqlJitPackages.postgis.doc (postgresql15JitPackages.postgis.doc)
  • postgresql15Packages.postgis
  • postgresql15Packages.postgis.doc
  • postgresql16JitPackages.postgis
  • postgresql16JitPackages.postgis.doc
  • postgresql16Packages.postgis
  • postgresql16Packages.postgis.doc

@nialov
Copy link
Contributor

nialov commented Jan 14, 2024

Result of nixpkgs-review pr 276348 run on x86_64-linux 1

1 test built:
  • nixosTests.postgis

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/week-in-geospatial-team/37035/3

@marsam
Copy link
Contributor

marsam commented Jan 19, 2024

@GrahamcOfBorg eval

@ofborg ofborg bot requested a review from das-g January 19, 2024 04:32
@imincik imincik merged commit 908f931 into master Jan 19, 2024
27 of 28 checks passed
@imincik imincik deleted the postgis-gdal-minimal branch January 19, 2024 09:09
imincik added a commit to imincik/geospatial-nix that referenced this pull request Jan 19, 2024
@imincik
pkgs(postgis): build with minimal gdal version
fdb0921 
Nixpkgs PR:
NixOS/nixpkgs#276348
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/week-in-geospatial-team/37035/6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marsam marsam marsam approved these changes

@thoughtpolice thoughtpolice Awaiting requested review from thoughtpolice thoughtpolice is a code owner

@nh2 nh2 Awaiting requested review from nh2

@r-burns r-burns Awaiting requested review from r-burns

@sikmir sikmir Awaiting requested review from sikmir

@nialov nialov Awaiting requested review from nialov

@MarcWeber MarcWeber Awaiting requested review from MarcWeber

@willcohen willcohen Awaiting requested review from willcohen

@das-g das-g Awaiting requested review from das-g

Assignees

@imincik imincik

Labels
6.topic: nixos 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 11.by: package-maintainer 12.approvals: 1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@imincik @das-g @nixos-discourse @nialov @marsam @delroth