DM-15465: Clean up utils usage with a view to supporting test running in pull requests

.github/workflows/build.yaml

@@ -0,0 +1,47 @@
+name: build_and_test
+
+on:
+  - push
+  - pull_request
+
+jobs:
+  build_and_test:
+    # runs-on: ubuntu-latest
+    # Need >= 20.04 for modern sqlite. Can switch to latest when
+    # github change
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.7
+
+      - name: Install sqlite
+        run: sudo apt-get install sqlite libyaml-dev
+
+      # It seems like pip install of requirements for a URL package
+      # does not install those dependencies
+      - name: Install sphgeom dependencies
+        run: pip install -r https://raw.githubusercontent.com/lsst/sphgeom/master/requirements.txt
+
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+
+      # We have two cores so we can speed up the testing with xdist
+      - name: Install xdist and openfiles
+        run: pip install pytest-xdist pytest-openfiles
+
+      - name: Build and install
+        run: pip install -v .
+
+      - name: Run tests
+        run: pytest -r a -v -n 3 --open-files
+
+      - name: Install documenteer
+        run: pip install documenteer[pipelines]
+
+      - name: Build documentation
+        working-directory: ./doc
+        run: package-docs build

.github/workflows/mypy.yaml

@@ -19,9 +19,6 @@ jobs:
       - name: Install
         run: pip install mypy
 
-      - name: Ensure LSST module root
-        run: touch python/lsst/__init__.py
-
       - name: Change to source directory and run mypy
         working-directory: ./python
         run: mypy lsst

.gitignore

@@ -18,3 +18,6 @@ bin/*
 .idea/
 .vscode/
 mypy.log
+build/*
+dist/*
+python/lsst/utils/*

.yamllint.yaml

@@ -6,7 +6,7 @@ ignore: |
 rules:
   document-start: {present: false}
   line-length:
-    max: 80
+    max: 132
     allow-non-breakable-words: true
     allow-non-breakable-inline-mappings: true
     ignore: |

doc/lsst.daf.butler/configuring.rst

@@ -8,7 +8,7 @@ The main sections of the YAML file are handled separately by each sub configurat
 Each config specialization, registry, schema, storage class, composites, and dimensions knows the name of the key for its own section of the configuration and knows the names of files providing overrides and defaults for the configuration.
 Additionally, if the sub configuration contains a ``cls`` key, that class is imported and an additional configuration file name can be provided by asking the class for its defaultConfigFile  property.
 All the keys within a sub configuration are processed by the class constructed from ``cls``.
-The primary source of default values comes from ``$DAF_BUTLER_DIR/config`` –- this directory contains YAML files matching the names specified in the sub config classes and also can include names specified by the corresponding component class (for example `~lsst.daf.butler.datastores.posixDatastore.PosixDatastore`  specifies that configuration should be found in ``datastores/posixDatastore.yaml``.
+The primary source of default values comes from the ``config`` resource (accessed using `pkg_resources` and package ``lsst.daf.butler``) –- this directory contains YAML files matching the names specified in the sub config classes and also can include names specified by the corresponding component class (for example `~lsst.daf.butler.datastores.posixDatastore.PosixDatastore`  specifies that configuration should be found in ``datastores/posixDatastore.yaml``.
 There are additional search paths that can be included when a config object is constructed:
 
 1. Explicit list of directory paths to search passed into the constructor.
@@ -22,7 +22,7 @@ To construct a Butler configuration object (`~lsst.daf.butler.ButlerConfig`) fro
   Shell variables will be expanded.
   The contents of these files will then be inserted into the configuration at the same hierarchy as the ``includeConfigs`` directive, with priority given to the values defined explicitly in the parent configuration (for lists of include files later files overwrite content from earlier ones).
 * Each sub configuration class is constructed by supplying the relevant subset of the global config to the component Config constructor.
-* A search path is constructed by concatenating the supplied search path, the environment variable path (``$DAF_BUTLER_CONFIG_PATH``), and the daf_butler config directory (``$DAF_BUTLER_DIR/config``).
+* A search path is constructed by concatenating the supplied search path, the environment variable path (``$DAF_BUTLER_CONFIG_PATH``), and the daf_butler config directory (within the ``lsst.daf.butler`` package resource).
 * Defaults are first read from the config class default file name (e.g., ``registry.yaml`` for `~lsst.daf.butler.Registry`, and ``datastore.yaml`` for `~lsst.daf.butler.Datastore`) and merged in priority order given in the search path.
   Every file of that name found in the search path is read and combined with the others.
 * Then any ``cls``-specific config files are read, overriding the current defaults.

python/lsst/__init__.py

@@ -0,0 +1,2 @@
+import pkgutil
+__path__ = pkgutil.extend_path(__path__, __name__)

python/lsst/daf/butler/_butlerConfig.py

@@ -22,10 +22,15 @@
 """
 Configuration classes specific to the Butler
 """
+from __future__ import annotations
 
 __all__ = ("ButlerConfig",)
 
+import copy
 import os.path
+from typing import (
+    Optional
+)
 
 from .core import (
     ButlerURI,
@@ -68,14 +73,14 @@ class ButlerConfig(Config):
 
     def __init__(self, other=None, searchPaths=None):
 
-        self.configDir = None
+        self.configDir: Optional[ButlerURI] = None
 
         # If this is already a ButlerConfig we assume that defaults
         # have already been loaded.
         if other is not None and isinstance(other, ButlerConfig):
             super().__init__(other)
             # Ensure that the configuration directory propagates
-            self.configDir = other.configDir
+            self.configDir = copy.copy(other.configDir)
             return
 
         if isinstance(other, str):
@@ -101,11 +106,7 @@ def __init__(self, other=None, searchPaths=None):
         configFile = butlerConfig.configFile
         if configFile is not None:
             uri = ButlerURI(configFile)
-            if uri.scheme == "s3":
-                uri.updateFile("")
-                self.configDir = uri.geturl()
-            else:
-                self.configDir = os.path.dirname(os.path.abspath(configFile))
+            self.configDir = uri.dirname()
 
         # A Butler config contains defaults defined by each of the component
         # configuration classes. We ask each of them to apply defaults to