Completes the prototype implementation of Bazel persistent worker mod…

…e support in rules_haskell

Started in 132d0af. This commit imports worker's code and sets everything to make
the persistent worker mode available to a regular `rules_haskell` user.

The worker code lives in `tools/worker`.

The mode is supported in ghc_nixpkgs-toolchain only, at this point.

docs/haskell-use-cases.rst

@@ -337,3 +337,26 @@ There a couple of notes regarding the coverage analysis functionality:
   ``build`` / ``test`` performance.
 
 .. _hpc: https://hackage.haskell.org/package/hpc
+
+Persistent Worker Mode (experimental)
+-------------------------------------
+
+Bazel supports the special `persistent worker mode`_ when instead of calling the compiler
+from scratch to build every target separately, it spawns a resident process for this purpose
+and sends all compilation requests to it in the client-server fashion. This worker strategy
+may improve compilation times. We implemented a worker for GHC using GHC API.
+
+.. _persistent worker mode: https://blog.bazel.build/2015/12/10/java-workers.html
+
+To activate the persistent worker mode in ``rules_haskell`` the user adds a couple of lines
+in the ``WORKSPACE`` file to load worker's dependencies: ::
+
+  load("//tools:repositories.bzl", "rules_haskell_worker_dependencies")
+  rules_haskell_worker_dependencies()
+
+Then, the user will add ``--define use_worker=True`` in the command line when calling
+``bazel build`` or ``bazel test``.
+
+It is worth noting that Bazel's worker strategy is not sandboxed by default. This may
+confuse our worker relatively easily. Therefore, it is recommended to supply
+``--worker_sandboxing`` to ``bazel build`` -- possibly, via your ``.bazelrc.local`` file.

haskell/BUILD.bazel

@@ -72,3 +72,10 @@ haskell_toolchain_libraries(
     name = "toolchain-libraries",
     visibility = ["//visibility:public"],
 )
+
+config_setting(
+    name = "use_worker",
+    define_values = {
+        "use_worker": "True",
+    },
+)

haskell/defs.bzl

@@ -91,6 +91,12 @@ _haskell_common_attrs = {
         cfg = "host",
         default = Label("@rules_haskell//haskell:ghc_wrapper"),
     ),
+    "worker": attr.label(
+        default = None,
+        executable = True,
+        cfg = "host",
+        doc = "Experimental. Worker binary employed by Bazel's persistent worker mode. See docs/haskell-use-cases.rst",
+    ),
 }
 
 def _mk_binary_rule(**kwargs):
@@ -180,42 +186,11 @@ def _mk_binary_rule(**kwargs):
         **kwargs
     )
 
-haskell_test = _mk_binary_rule(test = True)
-"""Build a test suite.
-
-Additionally, it accepts [all common bazel test rule
-fields][bazel-test-attrs]. This allows you to influence things like
-timeout and resource allocation for the test.
-
-[bazel-test-attrs]: https://docs.bazel.build/versions/master/be/common-definitions.html#common-attributes-tests
-"""
+_haskell_test = _mk_binary_rule(test = True)
 
-haskell_binary = _mk_binary_rule()
-"""Build an executable from Haskell source.
-
-Example:
-  ```bzl
-  haskell_binary(
-      name = "hello",
-      srcs = ["Main.hs", "Other.hs"],
-      deps = ["//lib:some_lib"]
-  )
-  ```
+_haskell_binary = _mk_binary_rule()
 
-Every `haskell_binary` target also defines an optional REPL target that is
-not built by default, but can be built on request. The name of the REPL
-target is the same as the name of binary with `"@repl"` added at the end.
-For example, the target above also defines `main@repl`.
-
-You can call the REPL like this (requires Bazel 0.15 or later):
-
-```
-$ bazel run //:hello@repl
-```
-
-"""
-
-haskell_library = rule(
+_haskell_library = rule(
     _haskell_library_impl,
     attrs = dict(
         _haskell_common_attrs,
@@ -253,27 +228,83 @@ haskell_library = rule(
     ],
     fragments = ["cpp"],
 )
-"""Build a library from Haskell source.
 
-Example:
-  ```bzl
-  haskell_library(
-      name = "hello-lib",
-      srcs = glob(["src/**/*.hs"]),
-      src_strip_prefix = "src",
-      deps = [
-          "//hello-sublib:lib",
-      ],
-      reexported_modules = {
-          "//hello-sublib:lib": "Lib1 as HelloLib1, Lib2",
-      },
-  )
-  ```
+def _haskell_worker_wrapper(rule_type, **kwargs):
+    defaults = dict(
+        worker = select({
+            "@rules_haskell//haskell:use_worker": Label("@rules_haskell//tools/worker:bin"),
+            "//conditions:default": None,
+        }),
+    )
+    defaults.update(kwargs)
+
+    if rule_type == 1:
+        _haskell_binary(**defaults)
+    elif rule_type == 2:
+        _haskell_test(**defaults)
+    elif rule_type == 3:
+        _haskell_library(**defaults)
+
+def haskell_binary(**kwargs):
+    """Build an executable from Haskell source.
+
+    Example:
+    ```bzl
+    haskell_binary(
+    name = "hello",
+    srcs = ["Main.hs", "Other.hs"],
+    deps = ["//lib:some_lib"]
+    )
+    ```
 
-Every `haskell_library` target also defines an optional REPL target that is
-not built by default, but can be built on request. It works the same way as
-for `haskell_binary`.
-"""
+    Every `haskell_binary` target also defines an optional REPL target that is
+    not built by default, but can be built on request. The name of the REPL
+    target is the same as the name of binary with `"@repl"` added at the end.
+    For example, the target above also defines `main@repl`.
+
+    You can call the REPL like this (requires Bazel 0.15 or later):
+
+    ```
+    $ bazel run //:hello@repl
+    ```
+
+    """
+    _haskell_worker_wrapper(1, **kwargs)
+
+def haskell_test(**kwargs):
+    """Build a test suite.
+
+    Additionally, it accepts [all common bazel test rule
+    fields][bazel-test-attrs]. This allows you to influence things like
+    timeout and resource allocation for the test.
+
+    [bazel-test-attrs]: https://docs.bazel.build/versions/master/be/common-definitions.html#common-attributes-tests
+    """
+    _haskell_worker_wrapper(2, **kwargs)
+
+def haskell_library(**kwargs):
+    """Build a library from Haskell source.
+
+    Example:
+    ```bzl
+    haskell_library(
+    name = "hello-lib",
+    srcs = glob(["src/**/*.hs"]),
+    src_strip_prefix = "src",
+    deps = [
+    "//hello-sublib:lib",
+    ],
+    reexported_modules = {
+    "//hello-sublib:lib": "Lib1 as HelloLib1, Lib2",
+    },
+    )
+    ```
+
+    Every `haskell_library` target also defines an optional REPL target that is
+    not built by default, but can be built on request. It works the same way as
+    for `haskell_binary`.
+    """
+    _haskell_worker_wrapper(3, **kwargs)
 
 haskell_import = rule(
     _haskell_import_impl,

haskell/nixpkgs.bzl

@@ -83,7 +83,6 @@ haskell_toolchain(
     # hack in Nixpkgs.
     {locale_archive_arg}
     locale = {locale},
-    use_worker = {use_worker},
 )
         """.format(
             toolchain_libraries = toolchain_libraries,
@@ -96,7 +95,6 @@ haskell_toolchain(
             repl_ghci_args = repository_ctx.attr.repl_ghci_args,
             locale_archive_arg = "locale_archive = {},".format(repr(locale_archive)) if locale_archive else "",
             locale = repr(repository_ctx.attr.locale),
-            use_worker = repository_ctx.attr.use_worker,
         ),
     )
 
@@ -122,7 +120,6 @@ _ghc_nixpkgs_haskell_toolchain = repository_rule(
         "locale": attr.string(
             default = "en_US.UTF-8",
         ),
-        "use_worker": attr.bool(),
     },
 )
 
@@ -176,14 +173,9 @@ def haskell_register_ghc_nixpkgs(
         locale = None,
         repositories = {},
         repository = None,
-        nix_file_content = None,
-        use_worker = False):
+        nix_file_content = None):
     """Register a package from Nixpkgs as a toolchain.
 
-    Args:
-      use_worker: This is a part of experimental support for the persistent worker mode.
-        It is not intended for production usage, yet.
-
     Toolchains can be used to compile Haskell code. To have this
     toolchain selected during [toolchain
     resolution][toolchain-resolution], set a host platform that
@@ -244,7 +236,6 @@ def haskell_register_ghc_nixpkgs(
         repl_ghci_args = repl_ghci_args,
         locale_archive = locale_archive,
         locale = locale,
-        use_worker = use_worker,
     )
 
     # toolchain definition.

haskell/private/cabal_wrapper.sh.tpl

@@ -95,7 +95,7 @@ cd - >/dev/null
 # There were plans for controlling this, but they died. See:
 # https://github.com/haskell/cabal/pull/3982#issuecomment-254038734
 library=($libdir/libHS*.a)
-if [[ -n ${library+x} ]]
+if [[ -n ${library+x} && -f $package_database/$name.conf ]]
 then
     mv $libdir/libHS*.a $dynlibdir
     sed 's,library-dirs:.*,library-dirs: ${pkgroot}/lib,' \

haskell/private/context.bzl

@@ -40,13 +40,16 @@ def haskell_context(ctx, attr = None):
     if hasattr(ctx.executable, "_ghc_wrapper"):
         ghc_wrapper = ctx.executable._ghc_wrapper
 
+    worker = getattr(ctx.executable, "worker", None)
+
     return HaskellContext(
         # Fields
         name = attr.name,
         label = ctx.label,
         toolchain = toolchain,
         tools = toolchain.tools,
         ghc_wrapper = ghc_wrapper,
+        worker = worker,
         package_ids = package_ids,
         src_root = src_root,
         package_root = ctx.label.workspace_root + ctx.label.package,

haskell/private/ghc_wrapper.sh

@@ -8,10 +8,9 @@ while IFS= read -r line; do compile_flags+=("$line"); done < $1
 
 # Detect if we are in the persistent worker mode
 if [ "$2" == "--persistent_worker" ]; then
-    compile_flags=("${compile_flags[@]:1}")  # remove ghc executable
-    # This is a proof-of-concept implementation, not ready for production usage:
-    # it assumes https://github.com/tweag/bazel-worker/ installed globally as ~/bin/worker
-    exec ~/bin/worker ${compile_flags[@]} --persistent_worker
+    # This runs our proof-of-concept implementation of a persistent worker
+    # wrapping GHC. Not ready for production usage.
+    exec ${compile_flags[@]} --persistent_worker
 else
     while IFS= read -r line; do extra_args+=("$line"); done < "$2"
     "${compile_flags[@]}" "${extra_args[@]}" 2>&1 \

haskell/protobuf.bzl

@@ -174,6 +174,7 @@ def _haskell_proto_aspect_impl(target, ctx):
         executable = struct(
             _ls_modules = ctx.executable._ls_modules,
             _ghc_wrapper = ctx.executable._ghc_wrapper,
+            worker = None,
         ),
         # Necessary for CC interop (see cc.bzl).
         features = ctx.rule.attr.features,

haskell/toolchain.bzl

@@ -23,7 +23,20 @@ def _run_ghc(hs, cc, inputs, outputs, mnemonic, arguments, params_file = None, e
         env = hs.env
 
     args = hs.actions.args()
-    args.add(hs.tools.ghc)
+    extra_inputs = []
+
+    # Detect persistent worker support
+    flagsfile_prefix = ""
+    execution_requirements = {}
+    tools = []
+    if hs.worker != None:
+        flagsfile_prefix = "@"
+        execution_requirements = {"supports-workers": "1"}
+        args.add(hs.worker.path)
+        tools = [hs.worker]
+    else:
+        args.add(hs.tools.ghc)
+        extra_inputs += [hs.tools.ghc]
 
     # Do not use Bazel's CC toolchain on Windows, as it leads to linker and librarty compatibility issues.
     # XXX: We should also tether Bazel's CC toolchain to GHC's, so that we can properly mix Bazel-compiled
@@ -57,8 +70,7 @@ def _run_ghc(hs, cc, inputs, outputs, mnemonic, arguments, params_file = None, e
     hs.actions.write(compile_flags_file, args)
     hs.actions.write(extra_args_file, arguments)
 
-    extra_inputs = [
-        hs.tools.ghc,
+    extra_inputs += [
         compile_flags_file,
         extra_args_file,
     ] + cc.files
@@ -73,15 +85,9 @@ def _run_ghc(hs, cc, inputs, outputs, mnemonic, arguments, params_file = None, e
     else:
         inputs += extra_inputs
 
-    # Detect persistent worker support
-    flagsfile_prefix = ""
-    execution_requirements = {}
-    if hs.toolchain.use_worker:
-        flagsfile_prefix = "@"
-        execution_requirements = {"supports-workers": "1"}
-
     hs.actions.run(
         inputs = inputs,
+        tools = tools,
         input_manifests = input_manifests,
         outputs = outputs,
         executable = hs.ghc_wrapper,

tools/repositories.bzl

@@ -0,0 +1,28 @@
+"""Workspace rules (tools/repositories)"""
+
+load("@rules_haskell//haskell:cabal.bzl", "stack_snapshot")
+
+def rules_haskell_worker_dependencies():
+    """Provide all repositories that are necessary for `rules_haskell`'s tools to
+    function.
+    """
+    excludes = native.existing_rules().keys()
+
+    if "rules_haskell_worker_dependencies" not in excludes:
+        stack_snapshot(
+            name = "rules_haskell_worker_dependencies",
+            packages = [
+                "base",
+                "bytestring",
+                "filepath",
+                "ghc",
+                "ghc-paths",
+                "microlens",
+                "process",
+                "proto-lens",
+                "proto-lens-runtime",
+                "text",
+                "vector",
+            ],
+            snapshot = "lts-14.1",
+        )

tools/worker/BUILD.bazel

@@ -0,0 +1,20 @@
+load("@rules_haskell//haskell:cabal.bzl", "haskell_cabal_binary")
+
+haskell_cabal_binary(
+    name = "bin",
+    srcs = glob(["**"]),
+    visibility = ["//visibility:public"],
+    deps = [
+        "@rules_haskell_worker_dependencies//:base",
+        "@rules_haskell_worker_dependencies//:bytestring",
+        "@rules_haskell_worker_dependencies//:filepath",
+        "@rules_haskell_worker_dependencies//:ghc",
+        "@rules_haskell_worker_dependencies//:ghc-paths",
+        "@rules_haskell_worker_dependencies//:microlens",
+        "@rules_haskell_worker_dependencies//:process",
+        "@rules_haskell_worker_dependencies//:proto-lens",
+        "@rules_haskell_worker_dependencies//:proto-lens-runtime",
+        "@rules_haskell_worker_dependencies//:text",
+        "@rules_haskell_worker_dependencies//:vector",
+    ],
+)