get_visual_stimuli_df should return correct information for images and gratings

[wbwakeman]

As a visual behavior analyst, I want to track start time, stop time, duration, and properties for images and gratings, so that I can do a complete analysis of a visual behavior experiment.

To do this, need to map information about gratings into the data structure that currently exists only for static images. Gratings properties are in the pkl file, and include:
spatial frequency (how wide is the bar)
phase: where in the wave does the bar begin +/- 0.25
rotation: +90/-90

• Change the gratings name in allensdk/brain_observatory/behavior/trials_processing.py to the convention of gratings_<angle>
• Add tests for the function get_trials in the trials_processing module (this function appears to be untested but a test file does exist)

Acceptance criteria:

• Demonstrate for an ophys behavior session, and a behavior-only session that stimulus_presentations table can be generated with valid data for image and grating stimulus presentations
• Creation of a pandas dataframe that can be presented to the visual behavior team that shows the correct naming convention for gratings stimuli

Repro steps:

from allensdk.brain_observatory.behavior.behavior_project_cache import BehaviorProjectCache as bpc
MANIFEST_PATH = "//allen/programs/braintv/workgroups/nc-ophys/visual_behavior/2020_cache/production_cache/manifest.json"
cache = bpc.from_lims(manifest = MANIFEST_PATH)
behavior_session_id = 983472077
session = cache.get_behavior_session_data(behavior_session_id)
session.trials

Stack trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/allen/aibs/technology/waynew/conda/allensdk_latest/lib/python3.7/site-packages/allensdk/brain_observatory/behavior/behavior_data_session.py", line 224, in trials
    self._trials = self.api.get_trials()
  File "/allen/aibs/technology/waynew/conda/allensdk_latest/lib/python3.7/site-packages/allensdk/internal/api/behavior_data_lims_api.py", line 326, in get_trials
    stimulus_presentations = self.get_stimulus_presentations()
  File "/allen/aibs/technology/waynew/conda/allensdk_latest/lib/python3.7/site-packages/allensdk/internal/api/behavior_data_lims_api.py", line 234, in get_stimulus_presentations
    data, stimulus_timestamps)
  File "/allen/aibs/technology/waynew/conda/allensdk_latest/lib/python3.7/site-packages/allensdk/brain_observatory/behavior/stimulus_processing.py", line 44, in get_stimulus_presentations
    stimulus_table = get_visual_stimuli_df(data, stimulus_timestamps)
  File "/allen/aibs/technology/waynew/conda/allensdk_latest/lib/python3.7/site-packages/allensdk/brain_observatory/behavior/stimulus_processing.py", line 229, in get_visual_stimuli_df
    draw_log_rising_edges = len(np.where(np.diff(stimuli['images']['draw_log'])==1)[0])
KeyError: 'images'

[wbwakeman]

FYI, grating image request now tracked in issue #1667

[kschelonka]

I spoke with @dougollerenshaw to clarify the requirements. He recommended bringing in @alexpiet and @matchings for their input.

For reference, the proposed implementation in the PR produces output for get_stimulus_metadata that looks like this (example has gratings only):

So the image_name encodes the orientation, and there are new columns added for "phase" and "frequency". These columns will be NaN for images.

From my notes, we discussed encoding the frequency, phase, and orientation in the image_name field. Doug had also mentioned using "horizontal" and "vertical" in the image_category. Can you all please confirm which format you would prefer?

[dougollerenshaw]

I just pulled down @isaak-willett 's current branch for testing. When I run the following:

# on branch https://github.com/AllenInstitute/AllenSDK/tree/GH-1666/feature/get_visual_stimuli_df_gratings
from allensdk.brain_observatory.behavior.behavior_data_session import BehaviorDataSession
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))

behavior_session_ids = [983472077, 818539440, 817861574]

session = {}
for bsid in behavior_session_ids:
    session[bsid] = BehaviorDataSession.from_lims(bsid)

session[behavior_session_ids[0]].stimulus_presentations.sample(10)

Here is the table that I get:

The parameters that are missing to fully describe the grating stimuli are spatial_frequency and phase.

@kschelonka mentioned the get_stimulus_metadata method above, which would include those parameters. The current implementation does not appear to include that method:

Assuming we have access to those parameters, this implementation seems like it would be sufficient to me to reconstruct the experiment.

@alexpiet or @matchings , care to weigh in?

[kschelonka]

@dougollerenshaw Hi Doug, get_stimulus_metadata is called by get_stimulus_presentations but is not included in the BehaviorDataSession api. So it could be propagated through from the results of BehaviorDataSession.stimulus_presentations.

I made a comment asking @alexpiet what the flow is for accessing the image metadata, since I agree with you that it seems not all the info is available easily. I think it might be get_stimulus_templates but I'm not sure. get_stimulus_templates will throw an error in this case because it assumes the presence of images key (which is missing when only gratings were shown).

FYI: if you want to import get_stimulus_metadata:
from allensdk.brain_observatory.behavior.stimulus_processing import get_stimulus_metadata

[pickles-bread-and-butter]

@dougollerenshaw sorry about that, the columns you're looking for will now be present in the stimulus_presentations dataframe. If you update the branch to the most recent commit you should see the data.

[alexpiet]

@dougollerenshaw @kschelonka I think having things easily available in the stimulus_presentations table is the most important for easy workflow.

[pickles-bread-and-butter]

@alexpiet that is currently how the branch implementation works, it will now have columns for phase and frequency where values will only exist for the gratings.

[pickles-bread-and-butter]

This PR is under the blocked category until primary stakeholders can review it.