-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't read from one file then save to another file #668
Comments
You can write an external link to another object by using the from datetime import datetime
from pynwb import NWBFile, NWBHDF5IO, TimeSeries, get_manager
manager = get_manager()
nwb = NWBFile(session_description='aa', identifier='aa', session_start_time=datetime.now())
ts1 = TimeSeries(name='timeseries1', data=[4, 5, 6],
unit='na1', rate=1.0, starting_time=0.0)
nwb.add_acquisition(ts1)
with NWBHDF5IO('test_append.nwb', mode='w', manager=manager) as io:
io.write(nwb)
with NWBHDF5IO('test_append.nwb', 'r', manager=manager) as f:
nwb_r = f.read()
nwb = NWBFile(session_description='aa', identifier='aa', session_start_time=datetime.now())
nwb.add_acquisition(nwb_r.acquisition['timeseries1'])
with NWBHDF5IO('test_append2.nwb', mode='w', manager=manager) as io:
io.write(nwb) What you describe- linking an entire from datetime import datetime
from pynwb import NWBFile, NWBHDF5IO, get_manager
manager = get_manager()
nwb = NWBFile(session_description='aa', identifier='aa', session_start_time=datetime.now())
with NWBHDF5IO('file1.nwb', mode='w', manager=manager) as io:
io.write(nwb)
with NWBHDF5IO('file1.nwb', 'r', manager=manager) as f:
nwb = f.read()
with NWBHDF5IO('file2.nwb', mode='w', manager=manager) as io:
io.write(nwb) This would instruct pynwb to write the entire file as an external link to another file. You are correct that this leads to an error, but is this really what you are trying to do? |
Hi @bendichter I think I'm a bit lost. Here's what I want to do: I have a file called test.nwb. I want to read that in, add some data or something, and then save it out to a file called test2.nwb so that I have two nwb files side by side, the first of which (test.nwb) is missing this added dataset. I don't want the second file to be a link to the first, I want it to have all the data replicated. Is this possible? Or do I have to use some other tool to duplicate the file before adding to one of them (would that work)? |
@VBaratham by default PyNWB will in this case create external links in
|
Thanks for the clarification, Oliver. We should add that to the tutorials |
@bendichter what do you think the best place for this is. Maybe in the modular storage tutorial: |
Yeah or maybe in advanced IO |
This has nothing to do with external links. |
@t-b in your example, you are opening the file in write mode but then reading from it, which is not allowed. We should check if that happens and throw a user-friendly error if so. If you change
|
@rly Thanks, granted this was sloopy. Next try:
|
@oruebel @bendichter from datetime import datetime
from dateutil.tz import tzlocal
from pynwb import NWBFile, NWBHDF5IO, get_manager, TimeSeries
import nwbext_ecog
import numpy as np
import os
manager = get_manager()
# Creates file 1
nwb_1 = NWBFile(session_description='session', identifier='1', session_start_time=datetime.now(tzlocal()))
# Add signal
X_data = np.zeros((10,1000))
timestamps = list(range(1000))
test_ts = TimeSeries(name='test_timeseries', data=X_data, unit='s', timestamps=timestamps)
nwb_1.add_acquisition(test_ts)
with NWBHDF5IO('file_1.nwb', mode='w') as io:
io.write(nwb_1)
# Open file 1
with NWBHDF5IO('file_1.nwb', 'r', manager=manager) as io_1:
nwb_1 = io_1.read()
# Open file 2
with NWBHDF5IO('file_2.nwb', mode='w', manager=manager) as io_2:
nwb_2 = NWBFile(session_description='session', identifier='2', session_start_time=datetime.now(tzlocal()))
for aux in nwb_1.acquisition: # Acquisition
nwb_2.add_acquisition(nwb_1.get_acquisition(aux))
io_2.write(nwb_2, link_data=False)
print(nwb_2.acquisition['test_timeseries'])
# Open file 2 after deleting file 1
with NWBHDF5IO('file_2.nwb', 'r', manager=manager) as io:
nwb_2 = io.read()
print(nwb_2.acquisition['test_timeseries'])
|
re: @oruebel 's comment earlier about adding hdmf-dev/hdmf#108 will be able to help with that, but this will take a little time to tackle. FYI: a full copy of a file is supported by |
EDIT @rly I am running into the same set of problems (container source, default linking) while attempting to carry out the same read -> modify -> write separately workflow. As a workaround, I can:
This is pretty gross. Is there a better way? What is the actual purpose of container_source if it is being reset on each write? If there is not a better way currently, when do you think these issues could be addressed? We are using this functionality (via workaround) to process data we hope to release in June and it would be great if we could have clean code in place to process those data. |
@NileGraddis would this https://pynwb.readthedocs.io/en/stable/tutorials/general/scratch.html#copying-an-nwb-file workflow work for you? |
Thanks for the quick reply! I think that is almost there, except that I would like the copy to be deep rather than shallow - we're using this in a processing pipeline which incrementally adds to an nwb file which ought then to stand alone. |
@NileGraddis if you want a full physical copy of the file, then you could just copy the file first and then open the copy. HDF5IO.copy_file can do this (which is useful if you need to resolve external links) or you could just do standard python
to copy the file. |
Copying the file is certainly another workaround, though it requires that the intended output is an h5 file on disk, which might not be true in the future! I suppose that the main concern I have about my workaround is
What is the intent behind |
Hi,
As long as the NWB object is in memory, everything is fine, but when I try to write it on disk, I get the same error with the |
hdmf-dev/hdmf#326 should address this issue |
It seems that this has been fixed, has it? |
If you use the latest PyNWB with the latest HDMF, then you can call the export method on Removal of objects from an in-memory NWBFile before exporting is not supported in PyNWB yet, but will be supported in an upcoming release. |
1) Bug
Loading an NWBFile from one file, then saving it to another causes
ValueError: Can't change container_source once set
to be thrownSteps to Reproduce
Environment
Please describe your environment according to the following bullet points.
Checklist
The text was updated successfully, but these errors were encountered: