This repository contains the code for analyzing the impact of space weather on active satellite systems. The analysis is performed by comparing solar and active satellite data to identify correlations between them.
- After cloning the repository, navigate to the root directory and download submodules recursively.
- Create a new Conda environment with Python 3.8 and activate it.
- In the project root, run the following command to install the required packages:
Polaris' configurations are located in the cfg
folder. For more information,
refer to
the Polaris ML Sandbox repository.
- From the root of the project, collect data and train on it with the following command:
polaris batch --config_file cfg/<satellite_config>.json
or fetch telemetry by manually specifying dates (polaris batch problem, cant specify dates in cfg file):
polaris fetch --cache_dir ./data/<sat_name>/cache --start_date 1970-01-01 --end_date 2024-03-25 <sat_name> ./data/<sat_name>/cache/normalized_frames.json
- This will download all data into the
./data/<sat_name>
folder, wheresat_name
is specified incfg/polaris.json
. - Build the graph using the following command:
polaris viz ./data/<sat_name>/cache/graph.json
- Detect anomalies in telemetry data using the
polaris behave
command:
polaris behave ./data/<sat-name>/cache/normalized_frames.json --output_file ./data/<sat_name>/cache/anomaly_analysis.json
- Visualize anomaly detection using the
polaris report
command:
polaris report ./data/<sat_name>/cache/anomaly_analysis.json
- (Optional) Convert the dependency graph to another file format using
the
polaris convert
command:
polaris convert ./data/<sat_name>/cache/graph.json ./data/<sat_name>/cache/graph.gexf
- (Optional) Information about the learning experiment with mlflow:
mlflow ui
- Loading and processing JSON data from SatNOGS Grafana Dashboard
- Converting timestamps to universal format
- Filtering data by specific years
- Calculating various types of correlations
- Plotting scatter plots for visualization
- Preprocessing solar and satellite data
- Merging datasets and performing correlation analysis
The script calculates several types of correlation coefficients between the
columns of the processed dataset. It uses the pandas.DataFrame.corr()
method
to compute the Pearson correlation coefficients, which measure the linear
relationship between two datasets. Additionally, it computes the
Kendall Tau and Spearman rank correlation coefficients, which can capture
nonlinear relationships.
For each pair of columns being compared, the script calculates the Pearson
correlation, Kendall Tau, and Spearman rank correlation coefficients. It also
performs a Chi-square test of independence to determine if there is a
significant association between the two variables. The chi-square statistic and
the corresponding p-value are computed using scipy.stats.chi2_contingency()
,
which provides insights into the likelihood that the observed data could have
occurred by chance under the assumption of no association.
Scatter plots are generated for each pair of columns to visualize the relationships between them. This helps in identifying trends and patterns in the data. Heatmaps of correlation matrices can also be used to present the correlation data in a more compact and informative manner.
An example of an anomaly in the data. An example of a data change over time.The script logs important events and warnings during execution. For instance, if a column is nearly constant and therefore not suitable for correlation analysis, a warning is logged. All outputs are formatted professionally, with variables enclosed in square brackets for clarity.
We are utilizing a modified and upgraded version
of Polaris ML, enabling us to
use Python 3.8
with the latest setup tools. This modification includes the
ability to
use polaris behave
and report
ing capabilities.
Additionally, Sun reports have been enabled, and some visualization styles have
been adjusted.
However, the graph functionality is still not fully operational. The graph can
be accessed by pressing Ctrl + /
, it does display vertex names
correctly, but coloring is not available. As a workaround, exporting to Gephi is the only
viable option at this time.
More about eXtreme Gradient Boosting
Pull requests and issues are welcome. For major changes, please open an issue first to discuss what you would like to change.