-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-25447: Add support for read-only components #319
Commits on Jul 1, 2020
-
Configuration menu - View commit details
-
Copy full SHA for 8bf3e17 - Browse repository at this point
Copy the full SHA 8bf3e17View commit details -
Configuration menu - View commit details
-
Copy full SHA for 965657e - Browse repository at this point
Copy the full SHA 965657eView commit details -
Add support for StorageClass read-only components
This necessitated a allComponents method to return the read/write components and read-only components together.
Configuration menu - View commit details
-
Copy full SHA for 1a77ec8 - Browse repository at this point
Copy the full SHA 1a77ec8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 669e3fb - Browse repository at this point
Copy the full SHA 669e3fbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 198b563 - Browse repository at this point
Copy the full SHA 198b563View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8405916 - Browse repository at this point
Copy the full SHA 8405916View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f5fd72 - Browse repository at this point
Copy the full SHA 4f5fd72View commit details -
Configuration menu - View commit details
-
Copy full SHA for 695aeef - Browse repository at this point
Copy the full SHA 695aeefView commit details -
Add new specialist formatters for MetricsExample
This allows us to test specialist formatter capabilities without having to expand the generic YAML and pickle formatters. It also allows us to test read parameters in formatters in addition to assemblers.
Configuration menu - View commit details
-
Copy full SHA for 3c63cf6 - Browse repository at this point
Copy the full SHA 3c63cf6View commit details -
Do not tell component formatter what component it is
For a disassembled component the formatter is only interested in reading the entire file, not extracting a component from the component.
Configuration menu - View commit details
-
Copy full SHA for d513d90 - Browse repository at this point
Copy the full SHA d513d90View commit details -
Enable true disassembly testing in low-level datastore tests
This enables disassembly testing with the new formatters without involving butler registry.
Configuration menu - View commit details
-
Copy full SHA for cbe8784 - Browse repository at this point
Copy the full SHA cbe8784View commit details -
Configuration menu - View commit details
-
Copy full SHA for faa4525 - Browse repository at this point
Copy the full SHA faa4525View commit details -
Change way parameters are handled if read-only component
We have to process the parameters before we extract the read-only component.
Configuration menu - View commit details
-
Copy full SHA for b3b4868 - Browse repository at this point
Copy the full SHA b3b4868View commit details -
Configuration menu - View commit details
-
Copy full SHA for eebade8 - Browse repository at this point
Copy the full SHA eebade8View commit details -
Add makeAllComponentDatasetTypes method
This creates dataset types for all the components.
Configuration menu - View commit details
-
Copy full SHA for 529533a - Browse repository at this point
Copy the full SHA 529533aView commit details -
No longer write component dataset types to registry
We no longer store components at the registry level so there is not much to be gained by registering the dataset types.
Configuration menu - View commit details
-
Copy full SHA for 19d36cd - Browse repository at this point
Copy the full SHA 19d36cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f7053f - Browse repository at this point
Copy the full SHA 6f7053fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d45183 - Browse repository at this point
Copy the full SHA 1d45183View commit details -
Configuration menu - View commit details
-
Copy full SHA for 08dc7b1 - Browse repository at this point
Copy the full SHA 08dc7b1View commit details -
Enable read-only component reading from disassembled components
This requires a new method on assembler to return which component should be used to calculate the read-only component.
Configuration menu - View commit details
-
Copy full SHA for c34118c - Browse repository at this point
Copy the full SHA c34118cView commit details -
Move where parameter validation occurs
Parameters for read-only components are problematic since it is not entirely clear whether the parameters should be applied to the component that is used to calculate the read-only component, or should be applied to the calculation of the read-only component itself. The complication is that assemblers must support the same parameters as formatters (otherwise in-memory datastore can not function) and also can apply parameters that the formatter did not understand. In the current implementation this means that the assembler can only see the final storage class and at that point parameters might not be relevant. As concrete examples. In the test suite I have added a "counter" read-only component that returns the number of elements in metrics.data. This means that the assembler sees an Integer storage class and integers aren't amenable to parameters. Instead the "slice" parameter is assumed to apply to the "data" component and then "counter" is calculated on the sliced "data". Another example is bbox in Exposure. This can be applied to the "image" component. Where should parameters go? To the calculation of the "image" or to the calculation of the bounding box (origin is the only one). Once the bounding box is created though the "origin" parameter has no meaning so "origin" is only relevant when passed to the "image" and can't be relevant to the assembler. Therefore read-only components for disassembled composites do not receive any parameters.
Configuration menu - View commit details
-
Copy full SHA for 6e4d81e - Browse repository at this point
Copy the full SHA 6e4d81eView commit details -
Raise exception if DatasetType has parent storage class but is not co…
…mponent Check that a component does have a parent but a composite does not. The parent storage class is not checked to determine if the component is defined by it.
Configuration menu - View commit details
-
Copy full SHA for 76c899d - Browse repository at this point
Copy the full SHA 76c899dView commit details
Commits on Jul 2, 2020
-
Add some tests for DatasetType parentStorageClass
This required changes to how pickling and deep copy worked. Also to simplify pickling the parentStorageClass is now also a positional argument.
Configuration menu - View commit details
-
Copy full SHA for 59a9f95 - Browse repository at this point
Copy the full SHA 59a9f95View commit details -
Add special placeholder parent storage class and update method
This allows pipeline definitions to set a temporary parent when analyzing the pipeline but then during execution update it with the real parent.
Configuration menu - View commit details
-
Copy full SHA for b02123d - Browse repository at this point
Copy the full SHA b02123dView commit details