Watchman on OS X doesn't build with FSEvents

[wmertens]

• Kernel: Darwin Wouts-MBP 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
• System: OS X 10.11.4
• Nix version: 1.11.2
• Nixpkgs version: "16.09pre79463.d88ae10"

Watchman uses kqueue instead of FSEvents, which makes it basically unusable. This can be verified by looking at the log which starts with Using watcher mechanism kqueue.

I traced it down to CoreServices not being provided as a build input, but when I try to do that I get these build errors:

watcher/fsevents.c:118:31: error: unknown type name 'CFFileDescriptorRef'
static void fse_pipe_callback(CFFileDescriptorRef fdref,
                              ^
watcher/fsevents.c:138:3: error: use of undeclared identifier 'CFFileDescriptorContext'
  CFFileDescriptorContext fdctx;
  ^
watcher/fsevents.c:139:3: error: use of undeclared identifier 'CFFileDescriptorRef'
  CFFileDescriptorRef fdref;
  ^
watcher/fsevents.c:152:28: error: use of undeclared identifier 'fdctx'; did you mean 'fsctl'?
  memset(&fdctx, 0, sizeof(fdctx));
                           ^~~~~
                           fsctl

To build, I use this nixpkgs/config.nix:

{ pkgs }:
{
    packageOverrides = pkgs: rec {
        watchman = pkgs.callPackage ./watchman.nix {
            inherit (pkgs.darwin.apple_sdk.frameworks) CoreServices;
            };
    };
}

and this watchman.nix:

{ stdenv, lib, config, fetchFromGitHub, autoconf, automake, pcre
, confFile ? config.watchman.confFile or null, CoreServices
}:

stdenv.mkDerivation rec {
  name = "watchman-${version}";

  version = "4.5.0";

  src = fetchFromGitHub {
    owner = "facebook";
    repo = "watchman";
    rev = "v${version}";
    sha256 = "0hyj7nbsm5mv8zq2fldd06f92a3wamavha20311518qv7q5jj72v";
  };

  buildInputs = [ autoconf automake pcre ]
    ++ stdenv.lib.optional stdenv.isDarwin CoreServices;

  configureFlags = [
      "--enable-lenient"
      "--enable-conffile=${if confFile == null then "no" else confFile}"
      "--with-pcre=yes"

      # For security considerations re: --disable-statedir, see:
      # https://github.com/facebook/watchman/issues/178
      "--disable-statedir"
  ];

  preConfigure = ''
    ./autogen.sh
  '';

  meta = with lib; {
    description = "Watches files and takes action when they change";
    homepage    = https://facebook.github.io/watchman;
    maintainers = with maintainers; [ cstrahan ];
    platforms   = with platforms; linux ++ darwin;
    license     = licenses.asl20;
  };
}

/cc @copumpkin or @shlevy any ideas on how to fix this build? @cstrahan I presume you're not using Darwin? Any others that may know?

[wmertens]

@pikajude perhaps? Looks like it's either not getting CoreFoundation or CoreFoundation is missing CFFileDescriptorContext?

[matthewbauer]

We're still using the 10.9 headers for CF which is at v855.17. It looks like it needs to be above 1070. That could be why it's not working. You could try forcing it to use the "CF_new" attribute although that might still be broken.

[wmertens]

@matthewbauer well when I try to use that, it tries to read CF/new.nix but that is not in the tree. I tried to add it but it quickly went over my head. Is there a branch for it somewhere?

[matthewbauer]

@wmertens You should look at the logs... It's very possible it just never go added. It's going to end up looking like the original with probably some updated patches. I think @copumpkin mainly responsible for most of the SDK stuff in Darwin.

[aflatter]

@matthewbauer @copumpkin seems like we're still at v855.17 of CF. Will this change at some point? I'm having trouble with Elixir's ecto library and the problem is probably the same.

[ghost]

Recreated using this derivation breaks on 10.12.6 (16G29):

environment setup steps

$ nix-channel --list      
nixpkgs https://nixos.org/channels/nixpkgs-unstable
$ nix-env -i watchman

actual reproduction

$ cd /tmp && mkdir foo
$ watchman watch-project foo 
$ touch foo/test

After the touch command, you should see something like the following in your terminal:

_path_files:467: too many open files in system: /dev/null
_path_files:467: too many open files in system: /dev/null

You have to run pkill watchman to free up the system.

[copumpkin]

So the fundamental issue here is that the public CF projects don't define CFFileDescriptorRef, probably due to overzealous redaction and nobody ever noticing before. Someone needs to file a bug with the swift project to request that their Foundation project (now where CF lives) include the missing header file, and then we'll be all set.

[ghost]

created the following: https://bugs.swift.org/browse/SR-6136

[ajbouh]

Any update here?