Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mapproxy: init at 1.12.0 #68628

Merged
merged 1 commit into from Feb 6, 2020
Merged

mapproxy: init at 1.12.0 #68628

merged 1 commit into from Feb 6, 2020

Conversation

@Rakesh4G
Copy link
Contributor

@Rakesh4G Rakesh4G commented Sep 13, 2019

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @CMCDragonkai

@Rakesh4G Rakesh4G requested a review from FRidh as a code owner Sep 13, 2019
@Rakesh4G
Copy link
Contributor Author

@Rakesh4G Rakesh4G commented Sep 13, 2019

The test cases are disabled because
Dependency list is huge.
https://github.com/mapproxy/mapproxy/blob/master/requirements-tests.txt
There are security issues with package Riak
#33876
#56480

@CMCDragonkai
Copy link
Member

@CMCDragonkai CMCDragonkai commented Oct 2, 2019

I've done some checks in the source code regarding the usage of libraries. I found that:

  • There's a MAPPROXY_LIB_PATH that can be used to add paths to C .so objects or dlls so that the system can load C libraries to bind to.
  • The 2 C libraries that it makes use of are: libproj and libgdal. These are not necessary if the 2 python fallbacks are available: pyproj and gdal.

So in the example derivation I pasted above, it just has pyproj and gdal.

When I tried building locally with nix-build:

{
  pkgs ? (import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/83e01aa5e48a9107f3b988c9d4c1c21d610626fb.tar.gz) {}),
  pythonPath ? "python37"
}:
  with pkgs;
  let
    python = lib.getAttrFromPath (lib.splitString "." pythonPath) pkgs;
  in
    python.pkgs.buildPythonPackage rec {
      pname = "MapProxy";
      version = "1.12.0";

      src = python.pkgs.fetchPypi {
        inherit pname version;
        sha256 = "622e3a7796ef861ba21e42231b49c18d00d75f03eaf3f01a2b7687be7568e2ec";
      };

      propagatedBuildInputs = with python.pkgs; [
        pillow
        pyyaml
        pyproj
        shapely
        gdal
        lxml
      ];

      doCheck = false;

      meta = with lib; {
        description = "MapProxy is an open source proxy for geospatial data";
        homepage = https://mapproxy.org/;
        license = licenses.asl20;
        maintainers = with maintainers; [ rakesh4g ];
      };
    }

Every works fine. But I use the callPackage pattern required by python-modules in Nixpkgs, I end up with this error:

nix-build ./default.nix --attr pkgs.python3Packages.mapproxy
»» ~/Projects/nixpkgs
 ♖ ./result/bin/mapproxy-util --version
Traceback (most recent call last):
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/bin/.mapproxy-util-wrapped", line 7, in <module>
    from mapproxy.script.util import main
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/script/util.py", line 28, in <module>
    from mapproxy.script.conf.app import config_command
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/script/conf/app.py", line 32, in <module>
    from .sources import sources
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/script/conf/sources.py", line 17, in <module>
    from mapproxy.srs import SRS
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/srs.py", line 26, in <module>
    from mapproxy.proj import Proj, transform, set_datapath
  File "/nix/store/3vbi2pfcf9bh0x535k78ln9p371xd7h5-python3.7-MapProxy-1.12.0/lib/python3.7/site-packages/mapproxy/proj.py", line 262, in <module>
    raise ImportError('could not find libproj or pyproj')
ImportError: could not find libproj or pyproj
@CMCDragonkai
Copy link
Member

@CMCDragonkai CMCDragonkai commented Oct 2, 2019

Why would using the call package pattern result in such a difference? Given that my out-of-tree default.nix uses an older nixpkgs hash, it just means setuptools is part of the package environment, whereas the current nixpkgs master doesn't have that.

Furthermore this is not the only problem. Even though my bin/mapproxy-util works, it fails to work as server due to probably the application considering that bin/mapproxy-util is a python script and trying to run it with python, but it fails with a syntax error because the bin/mapproxy-util is a wrapped shell script due to Nix.

»» ~/Projects/mapproxy/mymapproxy
 ♖ ../result/bin/mapproxy-util serve-develop mapproxy.yaml
[2019-10-02 13:22:18,621] mapproxy.config - INFO - reading: /home/cmcdragonkai/Projects/mapproxy/mymapproxy/mapproxy.yaml
[info]  * Running on http://127.0.0.1:8080/ (Press CTRL+C to quit)
[info]  * Restarting with reloader
  File "/nix/store/bfvw32p8c61jajjm7a8z98xbk1n6qiwz-python3.7-MapProxy-1.12.0/bin/mapproxy-util", line 2
    export PATH='/nix/store/kimimvhimclyfzlncpg36zjni3wn70nq-python3-3.7.3/bin:/nix/store/bfvw32p8c61jajjm7a8z98xbk1n6qiwz-python3.7-MapProxy-1.12.0/bin:/nix/store/rpayih9fb4ms2wjvmyn6kp7kmsizim49-python3.7-setuptools-40.8.0/bin:/nix/store/kp0g7z5h4kr6gx71jf9q0a9322p4jr5p-python3.7-numpy-1.16.2/bin:/nix/store/7vgnq00rxw9xh03920apdyj4ibgsmrix-gdal-2.4.0/bin'${PATH:+':'}$PATH
              ^
SyntaxError: invalid syntax

I've reported this problem on mapproxy: mapproxy/mapproxy#424

@CMCDragonkai
Copy link
Member

@CMCDragonkai CMCDragonkai commented Oct 2, 2019

@Rakesh4G Also this is more of an application, not a package. Maybe we should be changing it to buildPythonApplication? Can you try changing this PR structure to instead mimic dvc's style: #54530

Notice where the default.nix is placed. It's not in python-modules.

@ofborg ofborg bot requested a review from CMCDragonkai Oct 2, 2019
@ofborg ofborg bot removed the 6.topic: python label Oct 24, 2019
buildInputs = [
pillow
pyyaml
python3Packages.pyproj

This comment has been minimized.

@FRidh

FRidh Oct 24, 2019
Member

no need for $python3Packages here

This comment has been minimized.

@Rakesh4G

Rakesh4G Oct 30, 2019
Author Contributor

Removed.

};

prePatch = ''
substituteInPlace mapproxy/util/ext/serving.py --replace "args = [sys.executable] + sys.argv" "args = sys.argv"

This comment has been minimized.

@FRidh

FRidh Oct 24, 2019
Member

throwing away the location of the interpreter seems incorrect to me

This comment has been minimized.

@Rakesh4G

Rakesh4G Oct 30, 2019
Author Contributor

@FRidh , Please suggest what should be the alternate to this. This done to get rid of the problem mapproxy/mapproxy#424

Thanks in advance.

@sikmir
Copy link
Member

@sikmir sikmir commented Jan 2, 2020

Is there any progress on it?

@Rakesh4G
Copy link
Contributor Author

@Rakesh4G Rakesh4G commented Jan 7, 2020

Is there any progress on it?

I am still waiting on some guidance to alternate options due to
#68628 (comment)

@Rakesh4G
Copy link
Contributor Author

@Rakesh4G Rakesh4G commented Jan 14, 2020

This can not be merged as is.
With Latest pyproj mapproxy breaks.
details here:
mapproxy/mapproxy#417

So we might need to add pyproj version - 1.9.6, With this version Mapproxy works.
@CMCDragonkai @FRidh

@CMCDragonkai
Copy link
Member

@CMCDragonkai CMCDragonkai commented Jan 17, 2020

If pyproj 2 is way different from pyproj1, we might need to keep a pyproj1 around in a separate derivation.

@Rakesh4G
Copy link
Contributor Author

@Rakesh4G Rakesh4G commented Jan 18, 2020

If pyproj 2 is way different from pyproj1, we might need to keep a pyproj1 around in a separate derivation.

@CMCDragonkai , OK. I will create a new PR to add support for pyproj1 as separate derivation.

@Rakesh4G
Copy link
Contributor Author

@Rakesh4G Rakesh4G commented Jan 20, 2020

Added PR for pyproj1:
#78092

@Rakesh4G Rakesh4G force-pushed the formbay:mapproxy-python branch from 946ed7e to 4a8725a Feb 6, 2020
@Rakesh4G Rakesh4G mentioned this pull request Feb 6, 2020
1 of 10 tasks complete
Copy link
Contributor

@jonringer jonringer left a comment

also, please use an indent width of 2, otherwise the package looks good to me

@jonringer
Copy link
Contributor

@jonringer jonringer commented Feb 6, 2020

should be just 1 commit, please squash.

git reset HEAD^^
git add pkgs/
git commit --amend --no-edit
git push .. .. --force
@Rakesh4G
Copy link
Contributor Author

@Rakesh4G Rakesh4G commented Feb 6, 2020

should be just 1 commit, please squash.

git reset HEAD^^
git add pkgs/
git commit --amend --no-edit
git push .. .. --force

ok.

@Rakesh4G Rakesh4G force-pushed the formbay:mapproxy-python branch from b3572f1 to 458b19a Feb 6, 2020
@Rakesh4G Rakesh4G changed the title pythonPackages.mapproxy: init at 1.12.0 mapproxy: init at 1.12.0 Feb 6, 2020
@jonringer
Copy link
Contributor

@jonringer jonringer commented Feb 6, 2020

@GrahamcOfBorg build mapproxy

@Rakesh4G
Copy link
Contributor Author

@Rakesh4G Rakesh4G commented Feb 6, 2020

Hi @jonringer , sorry i made one mistake. it is supposed to be a buildPythonApplication not buildPythonPackage. I am just updating this in a minute.

@Rakesh4G Rakesh4G force-pushed the formbay:mapproxy-python branch from 458b19a to 5e9fcaf Feb 6, 2020
@jonringer
Copy link
Contributor

@jonringer jonringer commented Feb 6, 2020

@GrahamcOfBorg build mapproxy

Copy link
Contributor

@jonringer jonringer left a comment

LGTM
executables show usage
has tests 👍
upstream seems somewhat active

[2 built, 0.0 MiB DL]
https://github.com/NixOS/nixpkgs/pull/68628
1 package built:
mapproxy
@jonringer jonringer merged commit 21b11bf into NixOS:master Feb 6, 2020
15 of 16 checks passed
15 of 16 checks passed
mapproxy on x86_64-linux
Details
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
mapproxy on aarch64-linux Success
Details
mapproxy on x86_64-darwin Success
Details
@Rakesh4G Rakesh4G deleted the formbay:mapproxy-python branch Feb 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.