Merge pull request #73 from ixdat/xrdml

v0.2.3: Spectrum readers, spectroelectrochemistry refactor, CyclicVoltammogram.plot_cycles

.gitignore

@@ -4,7 +4,7 @@ development_scripts/*/*.png
 development_scripts/*/*.svg
 development_scripts/*/*.csv
 docs/source/figures/big_src/
-
+*.pkl
 
 
 # Byte-compiled / optimized / DLL files

NEXT_CHANGES.rst

@@ -12,15 +12,143 @@ links to relevant Issues, Discussions, and PR's on github with the following for
 
 `PR #XX <https://github.com/ixdat/ixdat/pulls/XX>`_
 
+
+For ixdat 0.2.3
+===============
+
 API Changes
 -----------
 
+measurement
+^^^^^^^^^^^
+- Added example usage to docstring of ``Measurement.select_values``
+
+readers
+^^^^^^^
+- Added an ``XRDMLReader`` (reader="xrdml") for xml-formatted XRD spectrum files from,
+  for example, Empyrion's software. Usage::
+
+    from ixdat import Spectrum
+    my_xrd = Spectrum.read("my_file.xrdml", reader="xrdml")
+    my_xrd.plot()
+
+- Added an ``AvantageAVGReader`` (reader="avantage") for avantage-exported spectrum files from,
+  for example, Thermo Fisher's K-alpha equipment. Usage::
+
+    from ixdat import Spectrum
+    my_xps = Spectrum.read("my_file.avg", reader="avantage")
+    my_xps.plot()
+
+- Added a ``QEXAFSDATReader`` (reader="qexafs") for .dat files exported by Diamond
+  Synchrotron's B18-Core beamline. These data files include X-Ray Absorption across
+  multiple detector elements, X-ray fluorescence, and diagnostic information such as primary
+  beam intensity. All the data is read together as a new object called a ``MultiSpectrum``.
+  If the ``read`` function is called with ``technique="XAS"``, a single spectrum with the
+  XAS data (the "FFIO" column) is returned::
+
+    from ixdat import Spectrum
+    multi_spectrum = Spectrum.read("my_file.dat", reader="qexafs")
+    my_xas = multi_spectrum["QexafsFFI0"]   # index retrieves a Spectrum from a MultiSpectrum
+    # OR
+    my_xas = Spectrum.read("my_file.dat", reader="qexafs", technique="XAS")
+    my_xas.plot()
+
+- A new reading method ``Spectrum.read_set()`` imports and appends multiple spectrum
+  files. It is thus similar to ``Measurement.read_set()``. The appended spectra are
+  returned as a ``SpectrumSeries``, which has a heat plot as its default ``plot()``.
+
+- Both ``Measurement.read_set()`` and the new ``Spectrum.read_set()`` allow the user
+  to specify a string that appears in the middle of the names of the files to be read
+  and appended, rather than just the beginning::
+
+    from ixdat import Spectrum
+    spectrum_series = Spectrum.read_set(
+       part="data/XAS_spectrum", suffix=".dat", technique="XAS"
+    )  # reads .dat files in the directory "data" with "XAS_spectrum" in their name
+    spectrum_series.plot()  # heat plot with time on x-axis and energy on y-axis
+
 techniques
 ^^^^^^^^^^
 
-``ECMSMeasurement.ecms_calibration_curve`` now supports data specification using a
-a selector. To do so, specify the section to use as numbers in the argument ``selector_list``,
-the counter defining the sections (if different from the default selector) in ``selector_name``,
-and the length of the steady-state period at the end of the pulse in ``t_steady_pulse``.
-This can be much more convenient than manually specifying ``tspans``.
-Implemented in #74.
+- ``CyclicVoltammogram`` now has a ``plot_cycles`` function. This plots all
+  the cycles in the cv color coded by cycle number, with a scale bar.
+
+- Refactoring of ``SpectroECMeasurement`` in order to generalize aspects of combining
+  time-resolved measurements with spectra taken simultaneously. This results in the
+  following classes:
+
+  - ``SpectroMeasurement(Measurement)`` for any time-resolved measurement data with spectra.
+  - ``SpectroECMeasurement(SpectroMeasurement, ECMeasurement)``, ``technique="SEC"``
+    for an EC measurement with spectra. By default plots the EC data on a lower panel and
+    the spectral data as a heat plot on the upper panel.
+  - ``ECXASMeasurement(SpectroECMeasurement)``, ``technique="EC-XAS"`` for an EC measurement
+    with XAS spectra.
+  - ``ECOpticalMeasurement(SpectroECMeasurement)``, ``technique=EC-Optical`` for an EC
+    measurement with optical spectroscopy. A reference spectrum is required, and can also
+    be set using a reference time or reference potential after the data is loaded. By
+    default, optical density = **-log(y/y0)**, with ``y=spectrum_series.y`` and
+    ``y0=reference_spectrum.y``, is plotted instead of raw data.
+
+  Before the refactor, ``ECOpticalMeasurement`` had been called ``SpectroECMeasurement``.
+
+- Addition of a ``Measurement`` and a ``SpectrumSeries`` gives a ``SpectroMeasurement``
+  or a subclass thereof determined by hyphenating the technique. For example::
+
+    from ixdat import Measurement, Spectrum
+
+    my_ec = Measurement.read("my_ec_data.mpt", reader="biologic")
+    print(my_ec, ",", my_ec.technique)
+    # ECMeasurement(...) , EC
+
+    my_xas = Spectrum.read_set("data/my_xas", suffix=".dat", technique="XAS")
+    print(my_xas, ",", my_ec.technique)
+    # SpectrumSeries(...) , XAS
+
+    my_ec_xas = my_ec + my_xas
+    print(my_ec_xas, ",", my_ec_xas.technique)
+    # ECXASMeasurement(...) , EC-XAS
+
+    my_ec_xas.plot()  # XAS data heat plot in top panel and EC data in bottom panel.
+   
+- ``ECMSMeasurement.ecms_calibration_curve`` now supports data specification using a
+  a selector. To do so, specify the section to use as numbers in the argument ``selector_list``,
+  the counter defining the sections (if different from the default selector) in ``selector_name``,
+  and the length of the steady-state period at the end of the pulse in ``t_steady_pulse``.
+  This can be much more convenient than manually specifying ``tspans``.
+  Implemented in #74.
+
+plotters
+^^^^^^^^
+
+- Plotters for spectroelectrochemistry were refactored in a similar way to the technique
+  classes (see above, under "techniques"). All ``SpectroECMeasurements`` come with plotters
+  with ``plot_measurement()`` (heat plot and EC data vs time) and ``plot_vs_potential()``
+  (heat plot and EC data vs potential). For ``ECOpticalMeasurements``, these plot
+  optical density rather than raw data on the heat plots.
+
+exporters
+^^^^^^^^^
+- The main .csv file exported from an ``ECOpticalMeasurement`` refers to its auxilliary
+  files as::
+
+    'spectrum_series' in file: 'exported_sec_spectra.csv'
+    'reference_spectrum' in file: 'exported_sec_reference.csv'
+
+  Files exported by ``SpectroECMeasurement`` in ixdat<0.2.3 will need these two lines modified, i.e.
+  "spectra"->"spectrum_series" and "reference"->"reference_spectrum", before they can
+  be imported. After this modification they can be read as before by
+  ``Measurement.read(path_to_main_csv_file, reader="ixdat")``.
+
+Debugging
+---------
+
+- ``PVMassSpecReader`` (reader="pfeiffer") now identifies columns as masses when they have
+  names assigned during the data acquisition (e.g. "32_amu_(Oxygen)" as a column name),
+  whereas before it would only have identified as masses columns which ended with "_amu".
+
+- ``ECPlotter.plot_measurement`` and ``ECMSPlotter.plot_measuremnt`` raise warnings instead
+  of ``SeriesNotFoundError``\ s if they can't find the requested
+  potential (``U_name``) or current (``J_name``).
+
+- ``SpectrumSeriesPlotter.heat_plot()`` now correctly orients its data. Previously it
+  had been flipped in the vertical direction.

README.rst

@@ -32,7 +32,7 @@ repository accompanying your breakthrough publication and advancing our field.
 
 Version
 -------
-This is the latest released version.
+This is the latest version.
 
 For changes up to this version, see `CHANGES.rst <https://github.com/ixdat/ixdat/blob/main/CHANGES.rst>`_
 
@@ -75,10 +75,10 @@ designed into every level.
      - Released
      - - msrh_sec: .csv file sets from Imperial College London's SEC system
    * - X-ray photoelectron spectroscopy (XPS)
-     - Future
+     - Development
      -
    * - X-ray diffraction (XRD)
-     - Future
+     - Development
      -
    * - Low-Energy Ion Scattering (LEIS)
      - Future

development_scripts/reader_testers/test_avantage_reader.py

@@ -0,0 +1,13 @@
+"""Sandbox script to aid in development and demo of the Avantage reader"""
+
+from pathlib import Path
+
+from ixdat import Spectrum
+
+path_to_file = (
+    Path.home() / "Dropbox/ixdat_resources/test_data/avantage" / "XPS Survey.avg"
+)
+
+spectrum = Spectrum.read(path_to_file, reader="avantage")
+
+spectrum.plot()

development_scripts/reader_testers/test_qexafs_reader.py

@@ -0,0 +1,40 @@
+"""Sandbox script for use in the development of the qexafs reader"""
+
+from pathlib import Path
+from ixdat import Spectrum, Measurement
+
+data_dir = Path(
+    r"C:\Users\scott\Dropbox\WORKSPACES\Caiwu\22E23_beamline_data\constant potential"
+)
+
+if True:
+    xas = Spectrum.read(
+        data_dir / "540117_IrO2_crys_0.60V_1.dat", reader="qexafs", technique="XAS"
+    )
+    xas.plot()
+
+if True:
+    multi_spec = Spectrum.read(
+        data_dir / "540117_IrO2_crys_0.60V_1.dat",
+        reader="qexafs",
+    )
+    ax = multi_spec["QexafsFFI0"].plot()
+    ax2 = ax.twinx()
+    multi_spec["I0"].plot(ax=ax2, color="r")
+    ax2.set_ylim([-1e6, 2e6])
+
+if True:
+    xas_series = Spectrum.read_set(
+        part=data_dir / "IrO2_crys", suffix=".dat", reader="qexafs", technique="XAS"
+    )
+    ax = xas_series.heat_plot()
+    ax.get_figure().savefig("xas_heat_plot.png")
+
+    ec = Measurement.read(data_dir / "IrO2_CA_0p60_C02.mpt", reader="biologic")
+    ec.plot()
+
+    ec_xas = ec + xas_series
+    ec_xas.tstamp += ec_xas.t[0]
+
+    axes = ec_xas.plot(xspan=[11200, 11300])
+

development_scripts/reader_testers/test_xrdml_reader.py

@@ -0,0 +1,15 @@
+"""Sandbox script to aid in development and demo of the XRDML reader"""
+
+from pathlib import Path
+
+from ixdat import Spectrum
+
+path_to_file = (
+    Path.home()
+    / "Dropbox/ixdat_resources/test_data/xrdml"
+    / "GI-XRD Path 2_1 omega 0p5 step 10s.xrdml"
+)
+
+spectrum = Spectrum.read(path_to_file, reader="xrdml")
+
+spectrum.plot()

src/ixdat/__init__.py

@@ -1,6 +1,6 @@
 """initialize ixdat, giving top-level access to a few of the important structures
 """
-__version__ = "0.2.2"
+__version__ = "0.2.3dev_qexafs"
 __title__ = "ixdat"
 __description__ = "The in-situ experimental data tool"
 __url__ = "https://ixdat.readthedocs.io"

src/ixdat/exporters/sec_exporter.py

@@ -55,13 +55,15 @@ def prepare_header_and_data(self, measurement, columns, tspan=None, time_step=No
             self.path_to_file.stem + "_spectra.csv"
         )
         measurement = measurement or self.measurement
-        self.header_lines.append(f"'spectra' in file: '{path_to_spectra_file.name}'\n")
+        self.header_lines.append(
+            f"'spectrum_series' in file: '{path_to_spectra_file.name}'\n"
+        )
         self.spectra_exporter.export(measurement.spectrum_series, path_to_spectra_file)
         path_to_reference_spectrum_file = self.path_to_file.parent / (
             self.path_to_file.stem + "_reference.csv"
         )
         self.header_lines.append(
-            f"'reference' in file: '{path_to_reference_spectrum_file.name}'\n"
+            f"'reference_spectrum' in file: '{path_to_reference_spectrum_file.name}'\n"
         )
         self.reference_exporter.export(
             measurement.reference_spectrum, path_to_reference_spectrum_file

src/ixdat/measurements.py

@@ -9,7 +9,6 @@
 also defines the base class for Calibration, while technique-specific Calibration
 classes will be defined in the corresponding module in ./techniques/
 """
-from pathlib import Path
 import json
 import numpy as np
 from .db import Saveable, PlaceHolderObject, fill_object_list
@@ -236,7 +235,15 @@ def read_url(cls, url, reader, **kwargs):
         return measurement
 
     @classmethod
-    def read_set(cls, path_to_file_start, reader, suffix=None, file_list=None, **kwargs):
+    def read_set(
+        cls,
+        path_to_file_start=True,
+        part=None,
+        suffix=None,
+        file_list=None,
+        reader=None,
+        **kwargs,
+    ):
         """Read and append a set of files.
 
         Args:
@@ -245,26 +252,23 @@ def read_set(cls, path_to_file_start, reader, suffix=None, file_list=None, **kwa
                 interpreted as the folder where the file are.
                 `Path(path_to_file).name` is interpreted as the shared start of the files
                 to be appended.
-            reader (str or Reader class): The (name of the) reader to read the files with
-            file_list (list of Path): As an alternative to path_to_file_start, the
-                exact files to append can be specified in a list
+            part (Path or str): A path where the folder is the folder containing data
+                and the name is a part of the name of each of the files to be read and
+                combined.
             suffix (str): If a suffix is given, only files with the specified ending are
                 added to the file list
+            file_list (list of Path): As an alternative to path_to_file_start, the
+                exact files to append can be specified in a list
+            reader (str or Reader class): The (name of the) reader to read the files with
             kwargs: Key-word arguments are passed via cls.read() to the reader's read()
                 method, AND to cls.from_component_measurements()
         """
-        base_name = None
-        if not file_list:
-            folder = Path(path_to_file_start).parent
-            base_name = Path(path_to_file_start).name
-            file_list = [f for f in folder.iterdir() if f.name.startswith(base_name)]
-            if suffix:
-                file_list = [f for f in file_list if f.suffix == suffix]
+        from .readers.reading_tools import get_file_list
 
+        file_list = file_list or get_file_list(path_to_file_start, part, suffix)
         component_measurements = [
             cls.read(f, reader=reader, **kwargs) for f in file_list
         ]
-
         measurement = None
         for meas in component_measurements:
             measurement = measurement + meas if measurement else meas
@@ -1128,9 +1132,6 @@ def select_values(self, *args, **kwargs):
         are those where the values match the provided criteria, i.e. the part of the
         measurement where `self[series_name] == value`
 
-        TODO: Testing and documentation with examples like those suggested here:
-            https://github.com/ixdat/ixdat/pull/11#discussion_r677324246
-
         Any series can be selected for using the series name as a key-word. Arguments
         can be single acceptable values or lists of acceptable values. In the latter
         case, each acceptable value is selected for on its own and the resulting
@@ -1141,6 +1142,20 @@ def select_values(self, *args, **kwargs):
         which is named by `self.selelector_name`. Multiple criteria are
         applied sequentially, i.e. you get the intersection of satisfying parts.
 
+        Examples of valid calls given a measurement `meas`:
+        ```
+        # to select where the default selector is 3, use:
+        meas.select_values(3)
+        # to select for where the default selector is 4 OR 5:
+        meas.select_values(4, 5)
+        # to select for where "cycle" (i.e. the value of meas["cycle"].data) is 4:
+        select_values(cycle=4)
+        # to select for where "loop_number" is 1 AND "cycle" is 3, 4, or 5:
+        meas.select_values(loop_number=1, cycle=[3, 4, 5])
+        ```
+        Note, a value name with a space in it (like "cycle number") can unfortunately
+        not be selected for with this method.
+
         Args:
             args (tuple): Argument(s) given without keyword are understood as acceptable
                 value(s) for the default selector (that named by self.sel_str)
@@ -1221,6 +1236,11 @@ def __add__(self, other):
         MS datasets, for example) and appending (sequential EC datasets). Either way,
         all the raw series (or their placeholders) are just stored in the lists.
         """
+        from .spectra import SpectrumSeries, add_spectrum_series_to_measurement
+
+        if isinstance(other, SpectrumSeries):
+            return add_spectrum_series_to_measurement(self, other)
+
         new_name = self.name + " AND " + other.name
         new_technique = get_combined_technique(self.technique, other.technique)