tensorflow-proto: build failure on NixOS

[grwlf]

Hi. tensorflow-proto from Hackage fails to build on NixOs with the following error

Using ld found on system at:
/nix/store/3k978nnp7ppjim34di40vwz5f95rsxhw-gcc-wrapper-5.4.0/bin/ld
No lhc found
No lhc-pkg found
No pkg-config found
Using strip version 2.28 found on system at:
/nix/store/x9v0yxy5iybp2m2ccqwqkvxgjy7zrw8f-binutils-2.28/bin/strip
Using tar found on system at:
/nix/store/4vzpc229l7yzzhpmvc14237pwck6y6r5-gnutar-1.29/bin/tar
No uhc found
building
protoc: callProcess: runInteractiveProcess: exec: does not exist (No such file
or directory)

(full log https://hydra.nixos.org/build/53182954/nixlog/2).
Could you please give a clue of what the protoc is? What package does it belong to?
Regards.

[blackgnezdo]

protoc is Google Protocol Buffers Compiler. Have you seen #168 which could use your review help ?

[srhb]

The issue is ultimately caused by cabal2nix (which automatically generates the Haskell package sets) not recognizing protoc as a build tool with a nix name.

It can be worked around by specifying the override:

tensorflow-proto = addBuildTool super.tensorflow-proto pkgs.protobuf;

... (and perhaps a few others in the same vein) in configuration-common.nix in the nixpkgs haskell modules.
A better solution would be to make cabal2nix recognize the dependency, probably.

[chrissze]

Hi Sarah (srhb) , i use NixOS and this file ~/.nixpkgs/config.nix to build a 'myHaskellEnv' , I also want to build tensorflow but i encounter the same missing compiler 'protoc' problem

1. do I need to create my own "configuration-common.nix" ?
2. I found sample file of configuration-common.nix and lib.nix in
   https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/haskell-modules
   after I copy them in my folder ~/.nixpkgs ; how do i import these 2 files in my ~/nixpkgs/config.nix?

thanks.

[srhb]

@szehk Here's a small self-contained example.

You can read about overlays in the nixpkgs manual in order to integrate it into your regular environment.

Please note that this is specifically using NixOS 17.09 HEAD (as of today) since the version of tensorflow-proto in unstable is incompatible with the newer proto-lens.

[jyp]

@srhb Thanks for that! Would you by any chance have succeeded with the tensorflow library itself? I tried

        tensorflow = self.haskell.lib.overrideCabal superHS.tensorflow (drv: { librarySystemDepends = [super.python36Packages.tensorflow]; });

but unfortunately the configuration phase still cannot locate the C bindings for tensorflow.

[srhb]

@jyp Sorry, haven't tried. Perhaps you can take a look at how the haskell tensorflow library uses the c bindings?

https://github.com/tensorflow/haskell/blob/master/tools/userchroot.nix

[jyp]

@srhb Thanks a bunch for the link. Additionally I had to use patchelf to point tensorflow to libs in the nix store. But then the test suite fails anyway:

ParseAllOps: [Failed]
ERROR: toEnum: unknown value for enum DataType: 22
CallStack (from HasCallStack):
  error, called at dist/build/autogen/Proto/Tensorflow/Core/Framework/Types.hs:207:16 in tensorflow-proto-0.1.0.0-GW6eAirpxxgQS0lqQzXnG:Proto.Tensorflow.Core.Framework.Types

         Test Cases  Total      
 Passed  0           0          
 Failed  1           1          
 Total   1           1          
Test suite FFITest: FAIL

[fkm3]

If you are seeing an unknown enum value, the issue is likely that you are building the haskell code against one version of tensorflow's .proto files and then linking against another (newer) version of tensorflow.

[fkm3]

A little more detail: The repo was just updated to tensorflow1.8, so if you've pulled the latests commits and updated the third_party/tensorflow submodule (which is where we read the .proto files from; git submodule update --init --recursive), then it should work (or at least fail at a later step) with whatever tensorflow version you are linking against.

[jyp]

@fkm3 Thanks! I was in fact building from the hackage uploads --- so there was definitely a mismatch. I'll try to build from the source repo.

[fkm3]

The nix code has been updated recently, so I assume this is fixed now. Re-open if not.