Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long waiting time when executing cache.get_session_table() function #1696

Open
NeoNeuron opened this issue Aug 30, 2020 · 1 comment
Open
Labels
braintv relates to Insitute BrainTV program bug external Issues submitted from external users neuropixels

Comments

@NeoNeuron
Copy link

Describe the bug
When executing cache.get_session_table() function, the download of corresponding *.csv file completed immediately. However, the get_session_table function would just hang there for minutes before it returned the DataFrame.

To Reproduce
https://allensdk.readthedocs.io/en/latest/_static/examples/nb/ecephys_data_access.html

import os
import shutil

import numpy as np
import pandas as pd

from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache
data_directory = '/local1/ecephys_cache_dir' # must be updated to a valid directory in your filesystem

manifest_path = os.path.join(data_directory, "manifest.json")
cache = EcephysProjectCache.from_warehouse(manifest=manifest_path)

sessions = cache.get_session_table()
probes = cache.get_probes()
channels = cache.get_channels()
units = cache.get_units()

Expected behavior
Return DataFrames of sessions, probes, channels, and units shortly after the cache download these four files:

  1. sessions.csv (7.8 kB)
  2. probes.csv (27.0 kB)
  3. channels.csv (6.6 MB)
  4. units.csv (51.4 MB)

Actual Behavior
The download of these four *.csv files completed immediately. However, the "get" function would just hang there for minutes before it returned the DataFrame.

Environment (please complete the following information):

  • OS & version: macOS 10.15.3
  • Python version: 3.7.8
  • AllenSDK version: 2.1.0

Additional context
To bypass this slow data fetch issue with cache, I manually downloaded *.nwb files from DANDI Archive, and use EcephysSession.from_nwb_path() function to import raw *.nwb files. However, it seems that I cannot manually import LFP data of specific probes from local *.nwb files.

Is there any api or function that can provide way to do this?

Many thanks!

@NeoNeuron NeoNeuron added the bug label Aug 30, 2020
@wbwakeman wbwakeman added the braintv relates to Insitute BrainTV program label Aug 31, 2020
@NeoNeuron
Copy link
Author

NeoNeuron commented Sep 1, 2020

Also, I tested cache module in under Python version.

Environment

  • OS: macOS 10.15.3
  • Python version: 3.7.4
  • AllenSDK version: 2.1.0
  • pynwb version: 1.3.2
  • hdmf version: 1.6.3

It looks around 13 mins to complete this line below.

sessions = cache.get_session_table()

And it downloaded all four *.csv files in my working directory. And after that, the rest three get functions completed around 1 second for each.

probes = cache.get_probes()
channels = cache.get_channels()
units = cache.get_units()

In short, as long as all four *.csv files existing in the data_directory, the get functions of cache module work well. Otherwise, it took way more time.

I think it might lead from the slow download speed for these *.csv files. Maybe it would be much easier if the download links of these csv files can be provided for manual download.

@wbwakeman wbwakeman added external Issues submitted from external users neuropixels labels Oct 15, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
braintv relates to Insitute BrainTV program bug external Issues submitted from external users neuropixels
Projects
None yet
Development

No branches or pull requests

2 participants