Skip to content

Spire Metadata Parser

Latest
Latest

Choose a tag to compare

View all tags
@TheAlgoDev TheAlgoDev released this 17 Mar 22:50
· 10 commits to main since this release
framework
a715eaf
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Spire Metadata Parser v1.0.0 Release

Release Date: March 17, 2025

Overview

The Spire Metadata Parser is a Python-based utility designed to streamline the processing of Sun SPIRE CSV files containing solar measurement metadata. The tool automates the extraction and transformation of key data fields and loads them into an SQLite database. This solution is tailored for national labs and solar researchers, making it easier to manage and analyze solar measurement data with coding experience.

Key Features

1. File System Operations

  • Recursive directory traversal to identify Spire data folders (YYMMDD format)
  • File type filtering (CSV/TXT) with configurable extensions
  • Optional filename extraction mode

2. Metadata Processing

  • CSV-to-DataFrame transformation with column normalization
  • Automatic filename injection for data provenance
  • Dynamic column header handling

3. Database Integration

  • SQLite database creation/connection management
  • Batch insertion with pandas.DataFrame.to_sql
  • Append-only write mode for incremental updates

Requirements

  • Python Version: Python 3.x
  • Dependencies:
    • Pandas
    • SQLite3 (built into Python)

Installation & Setup

Configuration

  1. Set file paths
    Modify these variables in the script: 
    # Raw Spire CSV data directory (input)
spire_data_path = "C:/path/to/raw/spire/data"  

# SQLite database storage directory (output)
database_path = "C:/path/to/database/storage"
Assets 2
Loading