smFRET-MFD measurement type

[AndersBarth]

I have opened the discussion in the google group: https://groups.google.com/d/msg/photon-hdf5/0_gBZm7miK0/17Nhgr3pAQAJ

I have also modified the code in hdf5.py to accomodate the new measurement type 'smFRET-MFD', however I seem to not have permissions to push my branch with the changes.

I have essentially added the smFRET-MFD measurement type to the valid measurement types list, and modified _check_photon_data_tables as attached below. I have separated the check for polarized/split detection from the check of the number of spectral channels.

If you give me access rights, I can push the changes into a separate branch an make a pull request.

Best,
Anders

def _check_photon_data_tables(ph_data, setup, norepeat=False, pool=None,
                              skip_measurement_specs=False, verbose=False):
    """Assert that the photon_data group follows the Photon-HDF5 specs.
    """
    _assert_has_field('timestamps', ph_data, verbose=verbose)
    _assert_has_field('timestamps_specs', ph_data, verbose=verbose)
    _assert_has_field('timestamps_unit', ph_data.timestamps_specs,
                      verbose=verbose)

    if 'measurement_specs' not in ph_data:
        if not skip_measurement_specs:
            # Called to print a warning
            _assert_has_field('measurement_specs', ph_data, mandatory=False,
                              verbose=verbose, norepeat=norepeat, pool=pool)
        return

    meas_specs = ph_data.measurement_specs
    msg = 'Missing "measurement_type" in "%s".' % meas_specs._v_pathname
    _assert_has_field('measurement_type', meas_specs, msg, verbose=verbose)

    meas_type = meas_specs.measurement_type.read().decode()
    if verbose:
        print('* Measurement type: "%s"' % meas_type)
    _assert_valid(meas_type in valid_meas_types,
                  msg='Unknown measurement type "%s"' % meas_type)

    # At this point we have a valid measurement_type
    # We will check (and raise an error) for any missing field.
    msg = '\nThis field is mandatory for "%s" data.' % meas_type
    kwargs = dict(msg_add=msg, verbose=verbose)

    # Read number of channels in each branch
    num_ch = dict(spectral=setup.num_spectral_ch.read(),
                  split=setup.num_split_ch.read(),
                  polarization=setup.num_polarization_ch.read())

    # Check for spectral channels
    if meas_type in ('smFRET', 'smFRET-usALEX', 'smFRET-nsALEX', 'smFRET-MFD'):
        _msg = ('%s measurement requires /setup/num_spectral_ch = 2 '
                '(not %d).' % (meas_type, num_ch['spectral']))
        _assert_valid(num_ch['spectral'] == 2, msg=_msg)
    if meas_type == 'smFRET-usALEX-3c':
        _msg = ('%s measurement requires /setup/num_spectral_ch = 3 '
                '(not %d).' % (meas_type, num_ch['spectral']))
        _assert_valid(num_ch['spectral'] == 3, msg=_msg)

    # Check for polarized detection channels
    if meas_type in ('smFRET', 'smFRET-usALEX', 'smFRET-nsALEX',
                     'smFRET-usALEX-3c'):
        num_polarization_ch = 1
    elif meas_type in ('smFRET-MFD'):
        num_polarization_ch = 2
    _msg = ('%s measurement requires /setup/num_polarization_ch = %d '
            '(not %d).' % (meas_type, num_polarization_ch,
                           num_ch['polarization']))
    _assert_valid(num_ch['polarization'] == num_polarization_ch, msg=_msg)

    # Check for split detection channels
    if meas_type in ('smFRET', 'smFRET-usALEX', 'smFRET-nsALEX',
                     'smFRET-MFD', 'smFRET-usALEX-3c'):
        num_split_ch = 1
    _msg = ('%s measurement requires /setup/num_split_ch = %d '
            '(not %d).' % (meas_type, num_split_ch, num_ch['split']))
    _assert_valid(num_ch['split'] == num_split_ch, msg=_msg)

    # handle case of no detectors array and no detectors_specs
    # which is possible when measurement_type is 'generic'
    if 'detectors' in ph_data:
        det_specs = meas_specs.detectors_specs
        # Check for spectral/split/polarization channels in detectors_specs
        for feature, nch in num_ch.items():
            if nch > 1:
                for i in range(nch):
                    _assert_has_field('%s_ch%d' % (feature, i + 1),
                                      det_specs, **kwargs)

    # Check presences of repetititon rate with pulsed lasers
    msg0 = "According to /setup/excitation_cw some lasers are pulsed.\n"
    if not all(setup.excitation_cw[:]):
        m = "However, the field '/setup/laser_repetition_rates' is missing."
        _assert_has_field('laser_repetition_rates', setup, msg_add=msg0 + m)
        m = ("However, the field 'measurement_specs/laser_repetition_rate' "
             "is missing.")
        _assert_has_field('laser_repetition_rate', meas_specs,
                          msg_add=msg0 + m)
    else:
        # All lasers are CW, check that there is no pulsed laser field
        msg = ('According to /setup/excitation_cw all lasera are CW.\n'
               'However, `%s` has the field `laser_repetition_rate`.')
        _assert_valid('laser_repetition_rate' not in setup, msg % 'setup')
        _assert_valid('laser_repetition_rate' not in meas_specs,
                      msg % 'measurement_specs')

    msg_cw = """
    According to /setup/excitation_alternated this measurement uses
    laser alternation and CW lasers. However, there is no alex_period
    field in measurement_specs. alex_period is mandatory in measurements
    using alternation and CW lasers."""
    if all(setup.excitation_cw[:]):
        if any(setup.excitation_alternated[:]):
            _assert_has_field('alex_period', meas_specs,
                              msg_add=dedent(msg_cw))
        else:
            # No alternated laser, check that no ALEX field is present
            msg0 = ('According to /setup/excitation_alternated '
                    'no laser is alternated.\n')
            msg1 = 'However, measurement_specs has the ALEX field `%s`.'
            fields = ['alex_period', 'alex_offset']
            fields += ['alex_excitation_period%d' % i for i in (1, 2, 3)]
            for field in fields:
                _assert_valid(field not in meas_specs, msg0 + msg1 % field)

    if 'nanotimes' in ph_data and 'lifetime' in setup:
        _assert_valid(setup.lifetime.read(),
                      'Lifetime is False but nanotimes are present.')

    if 'lifetime' in setup and setup.lifetime.read():
        msg = """\
        According to /setup/lifetime (=True) this file should be a
        TCSPC measurement. However /setup/excitation_cw says that all
        the laser sources are CW instead of pulsed. At least one source
        needs to be pulsed."""
        _assert_valid(not all(setup.excitation_cw.read()), msg=dedent(msg))
        _assert_has_field('nanotimes', ph_data, **kwargs)

        if 'nanotimes_specs' in ph_data:
            _assert_has_field('nanotimes_specs', ph_data, **kwargs)
            tcspc_specs_group = ph_data.nanotimes_specs
        else:
            _assert_has_field('detectors', setup, **kwargs)
            tcspc_specs_group = setup.detectors
        for name in ('tcspc_unit', 'tcspc_num_bins'):
            _assert_has_field(name, tcspc_specs_group, **kwargs)

    # us-ALEX fields
    if meas_type in ('smFRET-usALEX', 'smFRET-usALEX-3c'):
        msg = 'All lasers need to be CW in %s measurements.'
        _assert_valid(all(setup.excitation_cw[:]), msg=msg % meas_type)
        msg = 'All lasers need to be alternated in %s measurements.'
        _assert_valid(all(setup.excitation_alternated[:]),
                      msg=msg % meas_type)

    # ns-ALEX / PIE fields
    if meas_type in ('smFRET-nsALEX', 'smFRET-MFD'):
        msg = "All lasers need to be pulsed in %s measurements." % meas_type
        _assert_valid(all(~setup.excitation_cw[:]), msg=msg)
        _assert_has_field('lifetime', setup, **kwargs)
        _assert_valid(setup.lifetime.read(),
                      msg='%s requires lifetime = True.' % meas_type)

[tritemio]

@AndersBarth, thanks for the contribution!

To open a Pull Request, push the modifications to your own fork, preferably in a dedicated branch. Then open a pull request (using the github interface) from your fork to the original repo. The permission issue you are seeing is because you are trying to push directly to the official phconvert repository.

However, regarding adding the support to MFD, this should be supported by the new "generic" measurement type in Photon-HDF5 v0.5 (implemented in phconvert 0.8+). See last point in:

• http://photon-hdf5.readthedocs.io/en/0.5.dev/whats_new_ver05.html

The introduction of a "generic" measurement type was motivated by the combinatorial explosion of needed measurement types needed to support CW and pulsed lasers, polarization/no polarization, spectral channels/no spectral channel, tcspc acquisition/no tcspc etc. For the historical discussion see:

Photon-HDF5/photon-hdf5#34

In this framework, you describe a TCSPC measurement with 1 laser and 2 polarizations with:

/setup/excitation_cw = [False]
/setup/lifetime = True
/setup/num_spectral_ch = 1
/setup/num_split_ch = 1
/setup/num_polarization_ch = 2

/photon_data/measurement_specs/detectors_specs/
    polarization_ch1 = 0
    polarization_ch2 = 1

For 2 lasers, 2 spectral detection channels:

/setup/excitation_cw = [False, False]
/setup/lifetime = True
/setup/num_spectral_ch = 2
/setup/num_split_ch = 1
/setup/num_polarization_ch = 1

/photon_data/measurement_specs/detectors_specs/
    spectral_ch1 = 0
    spectral_ch2 = 1

And for 2 lasers, 2 spectral channels and 2 polarization channels (4 detectors) you would use:

/setup/excitation_cw = [False, False]
/setup/lifetime = True
/setup/num_spectral_ch = 2
/setup/num_split_ch = 1
/setup/num_polarization_ch = 2

/photon_data/measurement_specs/detectors_specs/
    polarization_ch1 = [0, 2]
    polarization_ch2 = [1, 3]
    spectral_ch1 = [0, 1]
    spectral_ch2 = [2, 3]

Please let me know if this covers all your needs.

[AndersBarth]

Dear Antonio,

of course the 'generic' type covers the needs. However, it might still be worth to provide inherent support for MFD smFRET experiments, as many labs are solely doing these types of experiments.

As an example, if I only want to support MFD experiments in my software, I now have to perform multiple checks to see if the data conforms. It would be more convenient to be able to check based on measurement_type.

[talaurence]

I agree. It is common enough to warrant a name (now that I understand the naming protocols betters – thanks for the clarification Antonino). So: back to the naming. How about smFRET-MFD-nsALEX? MFD is a rather general term for looking at multiple colors and polarizations with TCSPC information, and was used for smFRET measurements. We added the alternating lasers to those measurements with nsALEX, so I think that would be descriptive enough. PIE is a fine terminology, but I think we should stick with the first term used.

[tritemio]

Ok, I though you where not aware of the generic measurement type.

So the discussion is: adding ad-hoc measurement types for common measurements to make them easier to read. Can you spell the specs for the measurement types you would like to add?

I was thinking that maybe we can get the advantage of both words by enforcing consistency between measurement_type and /setup group. So, if your software decodes the setup group then you can as well ignore the measurement type and read any Photon-HDF5 file. But if you are interested only in one measurement type, then you can read measurement_type and be sure that all the needed fields in /setup and measurement_specs are present.

There is a problem though. What if somebody saves a generic measurement type with setup configuration corresponding to one existing measurement type. In this case a software relying on the measurement_type field only will not recognize the file.

[tritemio]

@talaurence, in your definition, is smFRET-MFD-nsALEX two interleaved lasers and two spectral channels and two polarizations (4 detectors)?

[talaurence]

Good point. Would it make sense for Photon-HDF5 to check if a file with a “generic” type is consistent with one of the “common measurements” and tag it accordingly? That would be more work on the developer’s side, but could even be a feature. On the other side, would Photon-HDF5 need to check if a given file is actually consistent with the given measurement_type? From: Antonino Ingargiola <notifications@github.com> Reply-To: Photon-HDF5/phconvert <reply@reply.github.com> Date: Thursday, March 15, 2018 at 9:15 AM To: Photon-HDF5/phconvert <phconvert@noreply.github.com> Cc: "Laurence, Ted A." <laurence2@llnl.gov>, Comment <comment@noreply.github.com> Subject: Re: [Photon-HDF5/phconvert] smFRET-MFD measurement type (#27) Ok, I though you where not aware of the generic measurement type. So the discussion is: adding ad-hoc measurement types for common measurements to make them easier to read. Can you spell the specs for the measurement types you would like to add? I was thinking that maybe we can get the advantage of both words by enforcing consistency between measurement_type and /setup group. So, if your software decodes the setup group then you can as well ignore the measurement type and read any Photon-HDF5 file. But if you are interested only in one measurement type, then you can read measurement_type and be sure that all the needed fields in /setup and measurement_specs are present. There is a problem though. What if somebody saves a generic measurement type with setup configuration corresponding to one existing measurement type. In this case a software relying on the measurement_type field only will not recognize the file. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#27 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIAwG4DfziE2mRl3FMcjN95DbV9SYIyzks5tepN5gaJpZM4Sr0Sj>.

[talaurence]

Yes From: Antonino Ingargiola <notifications@github.com> Reply-To: Photon-HDF5/phconvert <reply@reply.github.com> Date: Thursday, March 15, 2018 at 9:19 AM To: Photon-HDF5/phconvert <phconvert@noreply.github.com> Cc: "Laurence, Ted A." <laurence2@llnl.gov>, Mention <mention@noreply.github.com> Subject: Re: [Photon-HDF5/phconvert] smFRET-MFD measurement type (#27) @talaurence<https://github.com/talaurence>, in your definition, is smFRET-MFD-nsALEX two interleaved lasers and two spectral channels and two polarization (4 detectors)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#27 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AIAwG8EyyC8TEzfFAZWSXw2kkDUj4oGbks5tepRzgaJpZM4Sr0Sj>.

[tritemio]

@talaurence, we already do all the checks to assure that each declared measuremnt_type has all the needed fields. So this is a just a matter of being more explicit in the documentation. The idea of converting "generic" measurement type to a matching specific type is interesting. But, what would you call a measurement doing two-color coincidence detection but no FRET (Klenerman), smFRET? In principle, this measurement could be saved as "generic" to state that is different from smFRET even though the setup is identical.

An option is just adding a warning when saving a "generic" file corresponding to specific measurement type. A user can decide to ignore the warning or to correct the measurement type as suggested. We could also add a keyword in save_photon_hdf5 to automatically override the generic type with the specific one, but I don't think it is a common enough case to warrant a new keyword.

[tritemio]

@AndersBarth I had a cursory read of your code and it seems legit. Please send a PR from your own fork as explained in my first comment. Can you also provide a small test data file for testing the conversion? I don't have a strong opinion on the name of the new measurement_type, but we need to reach an agreement.