Added RSF writer and tests

[aaronjgirard]

Description

Added a method to write out RSF files and included tests for the code.

Checklist

I have (if applicable):

• referenced the GitHub issue this PR closes.
• documented the new feature with docstrings or appropriate doc page.
• included a test. See testing guidelines.
• your name has been added to the contributors page (docs/contributors.md).
• added the "ready_for_review" tag once the PR is ready to be reviewed.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (704d19f) 99.29% compared to head (cee50a9) 99.30%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #301   +/-   ##
=======================================
  Coverage   99.29%   99.30%           
=======================================
  Files          84       86    +2     
  Lines        6714     6771   +57     
=======================================
+ Hits         6667     6724   +57     
  Misses         47       47           

Flag	Coverage Δ
unittests	99.30% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[d-chambers]

A couple small things then should be good to merge when you are happy with it.

[aaronjgirard]

Thanks, those are good changes. I have gone ahead and made them. We still do not have test coverage for a file that is not int or float input (frequency, for example), so one line of code is not covered in the tests. Do you have an example test dataset with complex numbers, for example? Also one for integer data? *Aaron Girard*

…

On Wed, Nov 22, 2023 at 7:30 AM Derrick Chambers ***@***.***> wrote: ***@***.**** requested changes on this pull request. A couple small things then should be good to merge when you are happy with it. ------------------------------ In dascore/io/rsf/core.py <#301 (comment)>: > + if np.issubdtype(dtype, np.integer): + file_formt = 'data_format="native_int"' + elif np.issubdtype(dtype, np.floating): + file_formt = 'data_format="native_float"' + else: + raise ValueError("Data format is not integer or floating.") Since we now explicitly convert to float32 dtype I think these lines can just be file_format = "native_float" right? ------------------------------ In dascore/io/rsf/core.py <#301 (comment)>: > + Write a patch to RSF format. + + if no data_path is NOT specified, the header and binary will be + packed together + data_path needs to be bindata_file.rsf or /location/of/bindata_file.rsf + (NO '@') + + path needs to be hdr_file.rsf or /location/of/hdr_file.rsf + + spool needs to have a single patch in it So this will generate a nice API doc page, we need a parameter section: Parameters----------------spool The input spool to convert to rsf, must have exactly one patch. path Path to create the rsf filedata_path If data and rsf header information are to be separate, the location of the data file. Needs to be bindata_file.rsf or /location/of/bindata_file.rsf (no '@') Notes--------- Patch datatype is converted to float32 for compatibility with Madagascar (may be able to keep dytpe in the future) ------------------------------ In dascore/io/rsf/__init__.py <#301 (comment)>: > +Notes +----- +- delete empty section, or add some notes here if you have any. Maybe the version of Madagascar you used to test the output? ------------------------------ In dascore/io/rsf/core.py <#301 (comment)>: > + ######################## + # Add a way to read in rsf files. + # May need a way to get absolute time stamp, + # like hide it in header file when writing out an RSF file. + ######################## Is this resolved? ------------------------------ In tests/test_io/test_rsf/test_rsf.py <#301 (comment)>: > +"""Tests for RSF format.""" + +import os +from pathlib import Path + +import numpy as np + +import dascore as dc +from dascore.io.rsf import RSFV1 + + +class TestRsfWrite: + """testing the rSF write out function.""" + + def test_write_nopath(self, random_patch, tmp_path): + """Test write function.""" maybe: """test write function with no data path specified""" ------------------------------ In tests/test_io/test_rsf/test_rsf.py <#301 (comment)>: > + + def test_write_nopath(self, random_patch, tmp_path): + """Test write function.""" + spool = dc.spool(random_patch) + path = tmp_path / "test_hdrdata.rsf" + RSFV1().write(spool, path) + + assert path.exists() + test_data = random_patch.data.astype(np.float32) + dtype = np.dtype(test_data.dtype) + file_esize = dtype.itemsize + datasize = test_data.size * file_esize + assert os.path.getsize(path) >= datasize + + def test_write_path(self, random_patch, tmp_path): + """Test write function.""" "Test write function with separate data path" — Reply to this email directly, view it on GitHub <#301 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEX7T345WLZKY5O5RS7FCFLYFYEALAVCNFSM6AAAAAA7VI5VX2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTONBUGUZTAOJUGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[d-chambers]

Do you have an example test dataset with complex numbers, for example?

We can use patch.dft to generate one like this: complex_patch = random_patch.dft("time")

But I think we are not supporting that now? This line explicitly converts any data type to float 32. If this is what you want to do you can essentially remove lines 68-73 and just have file_formt = 'data_format="native_float"' which will remove the untested lines and make codecov happy.

[aaronjgirard]

But I think we are not supporting that now? This line explicitly converts any data type to float 32. If this is what you want to do you can essentially remove lines 68-73 and just have file_formt = 'data_format="native_float"' which will remove the untested lines and make codecov happy.

The only problem then is in line 72-73, if someone ever tries to export a complex valued data set it cannot export because patch = spool[0] in line 56 cannot export and indeed fails when the following is added in the test:

complex_patch = random_patch.dft("time")
spool = dc.spool(complex_patch)

[d-chambers]

The only problem then is in line 72-73, if someone ever tries to export a complex valued data set it cannot export because patch = spool[0] in line 56 cannot export and indeed fails when the following is added in the test:

This is a bug. I will open another issue.

The dtype conversion will probably fail if it is complex, but should work fine with ints (it will just cast the ints to floats, which is probably fine).

[d-chambers]

see #303.

Once we get that fixed we can either explicitly support the complex type or just raise an error if the array dtype is complex. You can check it with np.issubdtype(dtype, np.complexfloating)

[aaronjgirard]

it looks like #304 fixed the complex values to spool issue, so I changed the RSF write and the test. However, now there another issue for writing integers to spool.

import dascore as dc

random_patch = dc.get_example_patch()
int_patch = random_patch.data.astype(np.int32)
spool = dc.spool(int_patch) # this raises error: "msg = 'Could not get spool from: {obj}" 

[d-chambers]

I think that is actually the expected behavior. The problem is int_patch = random_patch.data.astype(np.int32) returns a numpy array (not a patch) with dtype int32, and dascore doesnt know how to make a spool from that.

Try this:

import numpy as np

import dascore as dc

int_patch = random_patch.new(data=np.ones_like(random_patch.data))