Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-25447: Add support for read-only components #319

Merged
merged 24 commits into from
Jul 2, 2020
Merged

Commits on Jul 1, 2020

  1. Configuration menu
    Copy the full SHA
    8bf3e17 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    965657e View commit details
    Browse the repository at this point in the history
  3. Add support for StorageClass read-only components

    This necessitated a allComponents method to return the
    read/write components and read-only components together.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    1a77ec8 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    669e3fb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    198b563 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8405916 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    4f5fd72 View commit details
    Browse the repository at this point in the history
  8. Remove getStoredItemInfo API

    No longer used.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    695aeef View commit details
    Browse the repository at this point in the history
  9. Add new specialist formatters for MetricsExample

    This allows us to test specialist formatter capabilities
    without having to expand the generic YAML and pickle
    formatters.  It also allows us to test read parameters
    in formatters in addition to assemblers.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    3c63cf6 View commit details
    Browse the repository at this point in the history
  10. Do not tell component formatter what component it is

    For a disassembled component the formatter is only
    interested in reading the entire file, not extracting
    a component from the component.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    d513d90 View commit details
    Browse the repository at this point in the history
  11. Enable true disassembly testing in low-level datastore tests

    This enables disassembly testing with the new formatters
    without involving butler registry.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    cbe8784 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    faa4525 View commit details
    Browse the repository at this point in the history
  13. Change way parameters are handled if read-only component

    We have to process the parameters before we extract the
    read-only component.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    b3b4868 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    eebade8 View commit details
    Browse the repository at this point in the history
  15. Add makeAllComponentDatasetTypes method

    This creates dataset types for all the components.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    529533a View commit details
    Browse the repository at this point in the history
  16. No longer write component dataset types to registry

    We no longer store components at the registry level
    so there is not much to be gained by registering
    the dataset types.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    19d36cd View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    6f7053f View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    1d45183 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    08dc7b1 View commit details
    Browse the repository at this point in the history
  20. Enable read-only component reading from disassembled components

    This requires a new method on assembler to return which component
    should be used to calculate the read-only component.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    c34118c View commit details
    Browse the repository at this point in the history
  21. Move where parameter validation occurs

    Parameters for read-only components are problematic since it
    is not entirely clear whether the parameters should be applied
    to the component that is used to calculate the read-only
    component, or should be applied to the calculation of
    the read-only component itself.
    
    The complication is that assemblers must support the same
    parameters as formatters (otherwise in-memory datastore
    can not function) and also can apply parameters that
    the formatter did not understand. In the current implementation
    this means that the assembler can only see the final storage
    class and at that point parameters might not be relevant.
    
    As concrete examples. In the test suite I have added
    a "counter" read-only component that returns the
    number of elements in metrics.data.  This means that
    the assembler sees an Integer storage class and integers
    aren't amenable to parameters. Instead the "slice" parameter
    is assumed to apply to the "data" component and then
    "counter" is calculated on the sliced "data".
    
    Another example is bbox in Exposure. This can be applied
    to the "image" component. Where should parameters go?
    To the calculation of the "image" or to the calculation
    of the bounding box (origin is the only one). Once
    the bounding box is created though the "origin" parameter
    has no meaning so "origin" is only relevant when passed
    to the "image" and can't be relevant to the assembler.
    
    Therefore read-only components for disassembled
    composites do not receive any parameters.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    6e4d81e View commit details
    Browse the repository at this point in the history
  22. Raise exception if DatasetType has parent storage class but is not co…

    …mponent
    
    Check that a component does have a parent but a composite does not.
    The parent storage class is not checked to determine if the component
    is defined by it.
    timj committed Jul 1, 2020
    Configuration menu
    Copy the full SHA
    76c899d View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2020

  1. Add some tests for DatasetType parentStorageClass

    This required changes to how pickling and deep copy worked.
    Also to simplify pickling the parentStorageClass is now
    also a positional argument.
    timj committed Jul 2, 2020
    Configuration menu
    Copy the full SHA
    59a9f95 View commit details
    Browse the repository at this point in the history
  2. Add special placeholder parent storage class and update method

    This allows pipeline definitions to set a temporary parent
    when analyzing the pipeline but then during execution update
    it with the real parent.
    timj committed Jul 2, 2020
    Configuration menu
    Copy the full SHA
    b02123d View commit details
    Browse the repository at this point in the history