Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PR: Implement support for spectral data lazy loading and reshape cache. #840

Merged
merged 3 commits into from Aug 22, 2021

Conversation

KelSolaar
Copy link
Member

@KelSolaar KelSolaar commented Jul 31, 2021

This PR implements support for lazy loading of spectral data so that importing Colour is significantly faster, i.e. ~= 1.5sec vs
~5.5sec for 0.3.16 on my old Macbook Pro 2014.

Note: Some of the import improvements are also pertaining to some changes I made in the colour.plotting.tm3018 module.

image

image

The shipped spectral data is not instantiated until first used after full package import. The implementation uses the colour.utilities.LazyCaseInsensitiveMapping class that initially stores callables that are replaced by the data when an item is being requested:

    def __getitem__(self, item):
        import colour

        value = super(LazyCaseInsensitiveMapping, self).__getitem__(item)

        if callable(value) and hasattr(colour, '__disable_lazy_load__'):
            value = value()
            super(LazyCaseInsensitiveMapping, self).__setitem__(item, value)

        return value

A typical 0.3.16 dataset is as follows:

MSDS_CAMERA_SENSITIVITIES_DSLR = CaseInsensitiveMapping({
    'Nikon 5100 (NPL)':
        RGB_CameraSensitivities(
            DATA_CAMERA_SENSITIVITIES_DSLR['Nikon 5100 (NPL)'],
            name='Nikon 5100 (NPL)'),
    'Sigma SDMerill (NPL)':
        RGB_CameraSensitivities(
            DATA_CAMERA_SENSITIVITIES_DSLR['Sigma SDMerill (NPL)'],
            name='Sigma SDMerill (NPL)')
})

it is now defined as follows:

MSDS_CAMERA_SENSITIVITIES_DSLR = LazyCaseInsensitiveMapping({
    'Nikon 5100 (NPL)':
        partial(
            RGB_CameraSensitivities,
            DATA_CAMERA_SENSITIVITIES_DSLR['Nikon 5100 (NPL)'],
            name='Nikon 5100 (NPL)'),
    'Sigma SDMerill (NPL)':
        partial(
            RGB_CameraSensitivities,
            DATA_CAMERA_SENSITIVITIES_DSLR['Sigma SDMerill (NPL)'],
            name='Sigma SDMerill (NPL)')
})

In order to be able to compose datasets during import and avoid instantiation
to spectral data, the process is prevented by the colour.__disable_lazy_load__ attribute that is only set after the package is fully imported at the very end of colour.__init__.

This new mechanism also means that we have to update the signature of all the
objects that use a default spectral distribution or multi-spectral distribution
as a default argument otherwise, they are defined with the partial as a default argument which is unsuitable.

It is worth noting that this does not affect packages using Colour because it is Colour import time "magic" only.

I took that opportunity to introduce high-level wrappers, i.e. colour.colorimetry.reshape_sd and colour.colorimetry.reshape_msds, that handle alignement, extrapolation, interpolation and trimming while avoiding to have to make a copy systematically. The other benefit is that we can also cache the operation, and
a subsequent call for the same spectral data with exact same reshaping arguments will use the cached spectral data instead of regenerating, thus better performance.

A typical 0.3.16 definition is as follows:

def sd_to_XYZ_integration(
        sd,
        cmfs=MSDS_CMFS_STANDARD_OBSERVER['CIE 1931 2 Degree Standard Observer']
        .copy().trim(SPECTRAL_SHAPE_DEFAULT),
        illuminant=sd_ones(),
        k=None):

    if illuminant.shape != cmfs.shape:
        runtime_warning(
            'Aligning "{0}" illuminant shape to "{1}" colour matching '
            'functions shape.'.format(illuminant.name, cmfs.name))
        illuminant = illuminant.copy().align(cmfs.shape)

    if sd.shape != cmfs.shape:
        runtime_warning('Aligning "{0}" spectral distribution shape to "{1}" '
                        'colour matching functions shape.'.format(
                            sd.name, cmfs.name))
        sd = sd.copy().align(cmfs.shape)

It is now as follows:

def sd_to_XYZ_integration(sd, cmfs=None, illuminant=None, k=None):
    if cmfs is None:
        cmfs = reshape_msds(MSDS_CMFS_STANDARD_OBSERVER[_DEFAULT_MSDS_CMFS],
                            SPECTRAL_SHAPE_DEFAULT)

    if illuminant is None:
        illuminant = sd_ones()

    if illuminant.shape != cmfs.shape:
        runtime_warning(
            'Aligning "{0}" illuminant shape to "{1}" colour matching '
            'functions shape.'.format(illuminant.name, cmfs.name))
        illuminant = reshape_sd(illuminant, cmfs.shape)

    if sd.shape != cmfs.shape:
        runtime_warning('Aligning "{0}" spectral distribution shape to "{1}" '
                        'colour matching functions shape.'.format(
                            sd.name, cmfs.name))
        sd = reshape_sd(sd, cmfs.shape)

@KelSolaar KelSolaar added this to the v0.4.0 milestone Jul 31, 2021
@coveralls
Copy link

coveralls commented Jul 31, 2021

Coverage Status

Coverage decreased (-0.05%) to 99.642% when pulling d54f3c0 on feature/lazy_load into 0969a2f on develop.

@KelSolaar KelSolaar merged commit 799f3e0 into develop Aug 22, 2021
@KelSolaar KelSolaar deleted the feature/lazy_load branch August 28, 2021 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants