refactor: refactor and move the scripts around

[sahiljhawar]

This pull request reorganizes and refactors the EL-PASO package to improve code structure, clarify module responsibilities, and update documentation accordingly. The main changes involve moving utility functions to a new utils module, consolidating extraction routines under processing, and updating all references and imports throughout the codebase and documentation.

Module restructuring and refactoring:

• Moved extract_variables_from_files and ExtractionInfo from the top-level package to el_paso.processing.extraction, and updated all imports and documentation references accordingly. (el_paso/__init__.py, el_paso/processing/__init__.py, el_paso/processing/extraction.py, docs/API_reference/overview.md, docs/API_reference/processing/extraction.md, examples/Arase/arase_mepe.py, examples/Arase/get_arase_orbit_variables.py, examples/Arase/process_arase_xep_realtime.py, examples/Arase/process_pwe_densities.py, examples/ESA/process_ngrm_satellite.py, examples/GOES/process_goes_r_mps_high.py, examples/GOES/process_goes_realtime.py, docs/getting_started/basic_workflow.md) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19]
• Removed the top-level extract_variables_from_files and ExtractionInfo from the main API and __all__ exports. (el_paso/__init__.py)

Utilities module introduction:

• Moved convert_string_to_datetime from el_paso.processing.convert_string_to_datetime to the new el_paso.utils module, updated all code and documentation references, and removed the old file. (el_paso/utils.py, el_paso/processing/__init__.py, el_paso/processing/convert_string_to_datetime.py, docs/API_reference/overview.md, docs/API_reference/utilities/convert_string_to_datetime.md, examples/Arase/process_arase_xep_realtime.py, examples/ESA/process_ngrm_satellite.py, examples/GOES/process_goes_realtime.py) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]

Documentation updates:

• Updated all API reference and workflow documentation to reflect the new module structure for extraction and utility routines. (docs/API_reference/overview.md, docs/API_reference/processing/extraction.md, docs/API_reference/utilities/convert_string_to_datetime.md, docs/getting_started/basic_workflow.md) [1] [2] [3] [4] [5]

Minor improvements:

• Added missing copyright and contributor lines.
• Cleaned up code and removed unnecessary comments and files. [1] [2] [3]

These changes improve the clarity and maintainability of the codebase by grouping related functionality and ensuring the documentation is consistent with the code structure.

[Copilot]

Pull request overview

This PR reorganizes EL-PASO’s public API and internal module structure by moving extraction logic under el_paso.processing.extraction and relocating convert_string_to_datetime into a new el_paso.utils module, with broad updates across docs, examples, tutorials, and tests to match the new import paths.

Changes:

• Moved extraction entry points (ExtractionInfo, extract_variables_from_files) into el_paso.processing.extraction and updated call sites to use ep.processing.extract_variables_from_files.
• Moved convert_string_to_datetime to el_paso.utils.convert_string_to_datetime and updated references and API docs.
• Updated tutorials/notebooks and MkDocs navigation to reflect the new module layout.

Reviewed changes

Copilot reviewed 29 out of 32 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tutorials/5_magnetic_field_processing.ipynb	Updates extraction call path and refactors IRBEM library path handling; clears outputs.
tutorials/3_saving_strategies.ipynb	Updates extraction call path and log/output text; adjusts formatting.
tutorials/2_variable_class.ipynb	Updates extraction and datetime conversion paths; notebook content/output changes (includes an error output).
tutorials/1_download_data_and_extracting_variables.ipynb	Updates extraction call path and cleans notebook output/formatting.
tests/comparisons/test_mageph_rbsp.py	Updates extraction call path for comparisons test.
ruff.toml	Updates lint ignores and excludes.
mkdocs.yml	Updates API reference navigation to new module locations.
examples/VanAllenProbes/process_hope_electrons.py	Updates extraction call path.
examples/VanAllenProbes/process_efw_emfisis_density_combined.py	Updates extraction call path and datetime conversion path to ep.utils.
examples/VanAllenProbes/process_ect_combined.py	Updates extraction call path.
examples/PROBAV/process_ept_electron_fluxes.py	Updates extraction call path.
examples/POES/process_poes_ted.py	Updates extraction call path and modifies CLI loop error handling.
examples/POES/process_poes_meped.py	Updates extraction call path and modifies CLI loop error handling.
examples/minimal_example.py	Updates extraction call path.
examples/GOES/process_goes_realtime.py	Updates extraction call path and datetime conversion path to ep.utils.
examples/GOES/process_goes_r_mps_high.py	Updates extraction call path and removes inline PLR2004 suppression.
examples/ESA/process_ngrm_satellite.py	Updates extraction call path and datetime conversion path to ep.utils.
examples/Arase/process_pwe_densities.py	Updates extraction call path.
examples/Arase/process_arase_xep_realtime.py	Updates extraction call path and datetime conversion path to ep.utils.
examples/Arase/get_arase_orbit_variables.py	Updates extraction call path.
examples/Arase/arase_mepe.py	Updates extraction call path.
el_paso/utils.py	Adds convert_string_to_datetime to utils module.
el_paso/processing/extraction.py	Adds SPDX contributor line; extraction module is the new home for extraction API.
el_paso/processing/convert_string_to_datetime.py	Removes legacy datetime conversion module (function moved to el_paso.utils).
el_paso/processing/init.py	Re-exports extraction API from el_paso.processing.extraction; removes datetime conversion export.
el_paso/init.py	Stops re-exporting extract_variables_from_files; exposes utils and ExtractionInfo.
docs/getting_started/basic_workflow.md	Updates extraction docs link to new location.
docs/API_reference/utilities/convert_string_to_datetime.md	Points API docs to el_paso.utils.convert_string_to_datetime.
docs/API_reference/processing/extraction.md	Points API docs to el_paso.processing.extraction.
docs/API_reference/overview.md	Updates API overview links to the new extraction/utilities pages.

Comments suppressed due to low confidence (1)

tutorials/2_variable_class.ipynb:155

• This cell currently hard-codes irbem_lib_path = "../IRBEM/libirbem.so" and the notebook output contains an OSError because the shared library cannot be found. For tutorials, please avoid committing error outputs and use a portable default path (e.g. Path(ep.__file__).parent / "libirbem.so", or let the user pass the path) so the notebook runs out-of-the-box after installation.

    "from el_paso.processing.magnetic_field_utils.irbem import Coords\n",
    "\n",
    "irbem_lib_path = \"../IRBEM/libirbem.so\"\n",
    "\n",
    "xGDZ_arr = np.stack((variables[\"alt\"].get_data(), variables[\"lat\"].get_data(), variables[\"lon\"].get_data())).T\n",

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.