proj-data: init at 1.16.0 #280062

imincik · 2024-01-10T15:45:14Z

Description of changes

add initial version of proj-data package containing datum grid resource files for proj
package
add optional support for installing proj-data grid resource files in proj data directory

AFAIK, installation of proj-data resource files in proj directory is the only user-friendly way of enabling them for all programs using proj. Another alternative methods are:

setting PROJ_DATA variable to ${proj}/share/proj:${proj-data} (see proj-data package tests)
adding them to user home directories

which can be very tricky for other applications using proj in containers or multi-user environments and can be error-prone.
For more information, see: https://proj.org/en/9.3/resource_files.html

Installation of proj-data files to proj is optional, because proj-data repository size is large (can reach 1GB of size soon) and it avoids downloading it when building proj package version where proj-data is not needed.

Example usage

Example of building proj package containing proj-data grids provided by nz_linz:

$ nix-build -E 'with import ./. {}; proj.override { proj-data = proj-data.override { gridPackages = ["nz_linz"]; };  withProjData = true; }'

ls ./result/share/proj
CH                               nz_linz_nzgd2000-20130801.json           nz_linz_nzgd2000-ka20161114-grid02.tif   nz_linz_nzgd2000-ndm-grid01.tif
deformation_model.schema.json    nz_linz_nzgd2000-20140201.json           nz_linz_nzgd2000-ka20161114-grid03.tif   nz_linz_nzgd2000-ndm-grid02.tif
GL27                             nz_linz_nzgd2000-20150101.json           nz_linz_nzgd2000-ka20161114-grid04.tif   nz_linz_nzgd2000-si20030821-grid01.tif
ITRF2000                         nz_linz_nzgd2000-20160701.json           nz_linz_nzgd2000-ka20161114-grid05.tif   nz_linz_nzgd2kgrid0005.tif
ITRF2008                         nz_linz_nzgd2000-20171201.json           nz_linz_nzgd2000-ka20161114-grid06.tif   nz_linz_nzgeoid2009.tif
ITRF2014                         nz_linz_nzgd2000-20180701.json           nz_linz_nzgd2000-ka20161114-grid07.tif   nz_linz_nzgeoid2016.tif
nad27                            nz_linz_nzgd2000-c120100904-grid01.tif   nz_linz_nzgd2000-ka20161114-grid08.tif   nz_linz_ontpht1964-nzvd2016.tif
nad83                            nz_linz_nzgd2000-c220110222-grid01.tif   nz_linz_nzgd2000-ka20161114-grid09.tif   nz_linz_README.txt
nad.lst                          nz_linz_nzgd2000-c320110613-grid01.tif   nz_linz_nzgd2000-ka20161114-grid10.tif   nz_linz_stisht1977-nzvd2016.tif
nz_linz_auckht1946-nzvd2016.tif  nz_linz_nzgd2000-c420111223-grid01.tif   nz_linz_nzgd2000-ka20161114-grid11.tif   nz_linz_taraht1970-nzvd2016.tif
nz_linz_blufht1955-nzvd2016.tif  nz_linz_nzgd2000-ch20160214-grid01.tif   nz_linz_nzgd2000-ka20161114-grid12.tif   nz_linz_wellht1953-nzvd2016.tif
nz_linz_dublht1960-nzvd2016.tif  nz_linz_nzgd2000-cs20130721-grid01.tif   nz_linz_nzgd2000-lg20130816-grid01.tif   other.extra
nz_linz_duneht1958-nzvd2016.tif  nz_linz_nzgd2000-cs20130721-grid02.tif   nz_linz_nzgd2000-lg20130816-grid02.tif   proj.db
nz_linz_gisbht1926-nzvd2016.tif  nz_linz_nzgd2000-ds20090715-grid011.tif  nz_linz_nzgd2000-mq20041223-grid011.tif  proj.ini
nz_linz_lyttht1937-nzvd2016.tif  nz_linz_nzgd2000-ds20090715-grid012.tif  nz_linz_nzgd2000-mq20041223-grid012.tif  projjson.schema.json
nz_linz_motuht1953-nzvd2016.tif  nz_linz_nzgd2000-ds20090715-grid013.tif  nz_linz_nzgd2000-mq20041223-grid013.tif  README-PROJ-DATA.md
nz_linz_napiht1962-nzvd2016.tif  nz_linz_nzgd2000-ds20090715-grid014.tif  nz_linz_nzgd2000-mq20041223-grid014.tif  triangulation.schema.json
nz_linz_nelsht1955-nzvd2016.tif  nz_linz_nzgd2000-gs20071016-grid01.tif   nz_linz_nzgd2000-mq20041223-grid015.tif  world
nz_linz_nzgd2000-20000101.json   nz_linz_nzgd2000-ka20161114-grid01.tif   nz_linz_nzgd2000-mq20041223-grid016.tif

Things done

Add a 👍 reaction to pull requests you find important.

l0b0 · 2024-01-11T20:45:11Z

pkgs/development/libraries/proj/default.nix

@@ -58,6 +62,10 @@ stdenv.mkDerivation (finalAttrs: rec {

  doCheck = true;

+  postInstall = lib.optionalString withProjData ''
+    cp -R ${proj-data}/* $out/share/proj/


Suggested change

cp -R ${proj-data}/* $out/share/proj/

cp --recursive "${proj-data}" "$out/share/proj"

GNU cp supports all of -R, -r, --recursive, and the last one is the most obvious.

If I understand correctly $out/share/proj doesn't exist at this point, so we can copy the entire directory across. * doesn't match dotfiles by default, so this also ensures that we copy everything.

Quote variables for general safety.

GNU cp supports all of -R, -r, --recursive, and the last one is the most obvious.

Done. Thanks

If I understand correctly $out/share/proj doesn't exist at this point

Yes, it already exists and contains some files. No grid files should be hidden files, so I have no worries about not copying something what should be.

Quote variables for general safety.

I understand this point, but most of path variables in nixpkgs code (including the ones in this package) are not quoted. Nix store path can't contain any strange characters or spaces. I am leaving this unquoted for consistency.

(Feel free to close any discussions where I've 👍'ed your last reply - I don't have access to close discussions.)

l0b0 · 2024-01-11T21:50:43Z

pkgs/by-name/pr/proj-data/package.nix

+    runHook preInstall
+    shopt -s extglob
+
+    mkdir -p $out


Suggested change

mkdir -p $out

mkdir -p "$out"

See the comment about store path variables quoting above.

l0b0 · 2024-01-11T21:50:58Z

pkgs/by-name/pr/proj-data/package.nix

+    shopt -s extglob
+
+    mkdir -p $out
+    cp README.DATA $out/README-PROJ-DATA.md


Suggested change

cp README.DATA $out/README-PROJ-DATA.md

cp README.DATA "$out"/README-PROJ-DATA.md

See the comment about store path variables quoting above.

l0b0 · 2024-01-11T21:51:44Z

pkgs/by-name/pr/proj-data/package.nix

+    cp README.DATA $out/README-PROJ-DATA.md
+
+    for grid in ${builtins.toString gridPackages}; do
+      if [ ! -d $grid ]; then


Isn't this check redundant? The cp will fail if the directory doesn't exist.

If this check doesn't exist, the error message is quite confusing

> Running phase: installPhase > cp: missing destination file operand after '/nix/store/v3258iwkrrmh5iddli7sdzxgikiw3cyh-proj-data-1.16.0/' > Try 'cp --help' for more information. ```. I prefer to leave it as is.

Sounds good. In that case I'd recommend changing the check to if ! [[ -d "$grid" ]] - it won't produce a syntax error if anyone includes an empty grid package by accident.

l0b0 · 2024-01-11T21:52:10Z

pkgs/by-name/pr/proj-data/package.nix

+
+    for grid in ${builtins.toString gridPackages}; do
+      if [ ! -d $grid ]; then
+        echo "ERROR: Grid ($grid) does not exist."


This should probably go to stderr:

Suggested change

echo "ERROR: Grid ($grid) does not exist."

echo "ERROR: Grid ($grid) does not exist." >&2

Thanks, good point. Done

l0b0 · 2024-01-11T21:52:56Z

pkgs/by-name/pr/proj-data/package.nix

+        exit 1
+      fi
+
+      cp $grid/!(*.sh|*.py) $out/


Suggested change

cp $grid/!(*.sh|*.py) $out/

cp "$grid"/!(*.sh|*.py) "$out"/

That said, would it be possible to create a positive rather than negative glob to include the relevant files? Something like *.json *.tif *.tiff *.txt?

Good point, but it is very hard to create future-proof list of allowed file types. I would prefer to leave it as is for now.

l0b0 · 2024-01-11T22:01:19Z

pkgs/by-name/pr/proj-data/tests.nix

+runCommand "${pname}-tests" { meta.timeout = 60; }
+  ''
+    # add proj-data files to proj resources search path
+    export PROJ_DATA=${proj}/share/proj:${proj-data}


Should this possibly be set by the package itself?

I am not sure what is the best approach. We can wrap all executables with PROJ_DATA path, but it wouldn't help with dozens of other apps using proj as library (like QGIS for example).

I am not sure what is the best approach. We can wrap all executables with PROJ_DATA path, […]

Sounds good.

but it wouldn't help with dozens of other apps using proj as library (like QGIS for example).

Isn't that unrelated? We'd be making one thing work without breaking anything else, right?

l0b0 · 2024-01-11T22:04:22Z

pkgs/by-name/pr/proj-data/package.nix

+    let
+      # build custom package containing `nz_linz` grids
+      projDataWithGrid = finalAttrs.finalPackage.overrideAttrs (_: {
+        postInstall = "cp nz_linz/* $out/";


Shouldn't this override gridPackages instead?

Yes, I would like to, but finalPackage doesn't support override (it supports only overrideAttrs). Any suggestions are welcome.

I don't know Nix well enough to suggest anything. Feel free to ignore this, unless someone else has a suggestion.

l0b0 · 2024-01-11T22:05:48Z

pkgs/by-name/pr/proj-data/tests.nix

+  inherit (proj-data) pname;
+in
+runCommand "${pname}-tests" { meta.timeout = 60; }
+  ''


This test is using pipelines, so set -o pipefail might help.

Good point, done.

nixos-discourse · 2024-01-16T09:22:07Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/week-in-geospatial-team/37035/3

imincik · 2024-01-18T09:43:52Z

@dotlambda , are you OK with this PR ?

l0b0

https://github.com/NixOS/nixpkgs/pull/280062/files#r1458004097 is still applicable.

imincik · 2024-02-14T12:19:51Z

Consider using proj.[passthru.]withProjData instead of function parameter (see: #281739
for example).

imincik · 2024-02-22T15:25:23Z

I am replacing this PR with #290643 which is using better approach of adding grid packages to proj-data using withGridPackages passthru.

imincik force-pushed the proj-data-init branch from ff29980 to 971bee9 Compare January 10, 2024 15:50

ofborg bot added the 8.has: package (new) label Jan 10, 2024

ofborg bot requested review from nh2, r-burns, das-g, nialov, sikmir and willcohen January 10, 2024 17:04

ofborg bot added 11.by: package-maintainer 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Jan 10, 2024

imincik force-pushed the proj-data-init branch 3 times, most recently from d6f0028 to e8200c0 Compare January 11, 2024 09:21

imincik marked this pull request as ready for review January 11, 2024 09:45

imincik requested a review from l0b0 January 11, 2024 09:45

ofborg bot requested a review from dotlambda January 11, 2024 10:03

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 101-500 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 1 10.rebuild-linux: 1-10 labels Jan 11, 2024

l0b0 reviewed Jan 11, 2024

View reviewed changes

imincik added 2 commits January 18, 2024 10:42

proj-data: init at 1.16.0

0b89641

proj: add optional support to install proj-data resource files

6904ffe

imincik force-pushed the proj-data-init branch from e8200c0 to 6904ffe Compare January 18, 2024 09:42

imincik requested a review from l0b0 January 18, 2024 09:42

ofborg bot added 10.rebuild-linux: 11-100 and removed 10.rebuild-linux: 101-500 labels Jan 18, 2024

l0b0 suggested changes Jan 29, 2024

View reviewed changes

imincik marked this pull request as draft February 14, 2024 12:18

imincik closed this Feb 22, 2024

emilylange deleted the proj-data-init branch May 14, 2024 21:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proj-data: init at 1.16.0 #280062

proj-data: init at 1.16.0 #280062

imincik commented Jan 10, 2024 •

edited

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

l0b0 Jan 18, 2024

l0b0 Jan 11, 2024

imincik Jan 18, 2024

nixos-discourse commented Jan 16, 2024

imincik commented Jan 18, 2024

l0b0 left a comment

imincik commented Feb 14, 2024 •

edited

imincik commented Feb 22, 2024

	cp -R ${proj-data}/* $out/share/proj/
	cp --recursive "${proj-data}" "$out/share/proj"

	cp README.DATA $out/README-PROJ-DATA.md
	cp README.DATA "$out"/README-PROJ-DATA.md

	echo "ERROR: Grid ($grid) does not exist."
	echo "ERROR: Grid ($grid) does not exist." >&2

proj-data: init at 1.16.0 #280062

proj-data: init at 1.16.0 #280062

Conversation

imincik commented Jan 10, 2024 • edited

Description of changes

Example usage

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixos-discourse commented Jan 16, 2024

imincik commented Jan 18, 2024

l0b0 left a comment

Choose a reason for hiding this comment

imincik commented Feb 14, 2024 • edited

imincik commented Feb 22, 2024

imincik commented Jan 10, 2024 •

edited

imincik commented Feb 14, 2024 •

edited