Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Analysis of historical data from the Reverse Beacon Network
Fetching latest commit…
Cannot retrieve the latest commit at this time.
This directory contains various files and scripts associated with analysing historical data from the Reverse Beacon Network ("RBN"). These scripts should work on any modern operating system with the necessary packages installed. In particular, the scripts require one or more of the following be present: 1. bash, or a compatible shell (I use bash 4.2); 2. python (I use python 2.7.3) 3. R (I use 2.15.1) Note that if you maintain a copy of the entire RBN dataset locally using the download-rbn and convert-rbn scripts you need sufficient space for two copies of the file during the process of updating the dataset. Also, obviously, the files need to reside on a filesystem that supports files of sufficient size. As of this writing, the RBN dataset is around 50GB. Files of this size should not present a problem for any modern filesystem, but it does mean that you must be sure not to use an old filesystem with limits that do not allow for files of this size. If stored on a filesystem that supports compression, the actual amount of space used on the disk by the RBN dataset is much smaller because of the inherent redundancy in the dataset. On my desktop system the compression ratio is 3.95, meaning that the entire dataset requires only about 12GB of actual disk space; reading the entire dataset using, for example, extract-rbn.py, takes about 150 seconds. If you need the RBN data for only a particular date range, rather than downloading the entire dataset with the download-rbn and convert-rbn scripts, you might find it more convenient to download one of the files at: https://www.adrive.com/public/cQwkEB/rbn That directory contains compressed year-by-year files of the RBN data for the years from 2009 to 2016. These files are in exactly the same format as that produced by the convert-rbn script. The directory also contains a compressed summary file covering the same period; the format of this file is described in the blog post: http://drevans.blog.enginehousebooks.com/2017/01/ summary-file-for-rbn-data-2009-to-2016.html ---- compare-multiple-sigs Compare the signals from several stations against a single station, over a given time period, on a particular band and to a particular continent. Provides a 99% confidence interval for the difference between the means of each underlying signal distribution as compared to the signal distribution of the single target station. compare-multiple-sigs.R Takes output generated by call to compare-sigs.py within the bash script compare-multiple-sigs, and generates and displays the final plot. compare-sigs: Compare the signals between two stations over a given time period, on a particular band and to a particular continent. Provides an estimate of the difference between the means of the underlying signal distributions, the 95% and 99% confidence limits of that estimate, and the probability that one signal really is stronger than the other. compare-sigs.py: Used by compare-sigs. Scans the local RBN database and extracts the details of comparison reports, then outputs a list of the SNR differences for use by compare-sigs.R. compare-sigs.R Takes output from compare-sigs.py and plots it as a simple histogram. Also includes a calculation of the best estimate of the true difference in signal between the two compared stations given the input sample, and the 95% and 99% confidence limits of the estimate, if there are sufficient data. convert-rbn: Convert RBN data from the as-is format in the RBN zip files to the single-file CSV format used by these scripts. diurnal-posts-from-summary.R Generates the plot presented as an example of the use of the RBN summary file in the blog post: http://drevans.blog.enginehousebooks.com/2017/01/ summary-file-for-rbn-data-2009-to-2016.html. download-rbn: Download daily historical files from the RBN as-is, putting the data into files with friendly names. extract-rbn.py Extract data for a range of dates from a local RBN database, sending the result to standard output. get-rbn: Bring local database (rbndata.csv) up to date. plot-sig.R: Show reported signal strength over a given time period for a single station on all continents, for a particular band. plot-sig-5.R: Show reported signal strength over a given time period for a single station on all continents, for 80/40/20/15/10. plot-sig-9: Create and display plot that shows reported signal strength over a given time period for a single station on all continents, for 160/80/40/30/20/17/15/12/10 plot-sig-9.R: Used by plot-sig-9. Generate plot of reported signal strength over a given time period for a single station on all continents, for 160/80/40/30/20/17/15/12/10. plot-sig-zones-9: Create and display plot that shows reported signal strength over a given time period for a single station on the six CQ zones with the largest number of RBN posts of that station, for 160/80/40/30/20/17/15/12/10. plot-sig-zones-5.R: Show reported signal strength over a given time period for a single station on the six CQ zones with the largest number of RBN posts of that station, for 80/40/20/15/10. plot-sig-zones-9.R: Used by plot-sig-zones-9. Generate plot of reported signal strength over a given time period for a single station on the six CQ zones with the largest number of RBN posts of that station, for 160/80/40/30/20/17/15/12/10. posting-statistics.py Generates a summary file of annual, mensal and diurnal data that is useful for various kinds of summary analyses. rank-sig: Compare the signal of a station to the signals of others on the same continent over a given time period, on a particular band and to a particular continent. rank-sig.py: Used by rank-sig. Scans the local RBN database and provides as output a file that contains the following space-separated information over a given time period in each line: Epoch target-stn-SNR all-other-SNRs all-other-SNRs are the SNRs for all stations on the same continent as the target, as heard near to the epoch time by stations that have heard the target station. rank-sig.R: Takes output from rank-sig.py and plots it, showing both absolute SNR and percentile of SNR compared to other stations on the same continent. ---- These scripts are by N7DR, except that: (1) download-rbn is based on a script by HA5FTL; (2) convert-rbn is based on a script by HA5FTL. If you find these scripts useful, please take a moment to look at my web site, and consider reading one of my books: http://www.sff.net/people/N7DR If you have patches to improve these scripts, please let me know. In addition to these scripts, there is a brief note in the file "What the RBN reports" that describes what the RBN documentation means when the term "SNR" appears. Also, the directory "examples" contains some example output plots: examples/FT5ZM-20140125-20140211-by-zone-9.png Output from plot-sig-zones-9 command examples/K1N-20150201-20150215-by-continent-9.png Output from plot-sig-9 command examples/N7DR-10m-EU-20140215-20140216.png Output from the compare-multiple-sigs command examples/N7DR-K0EU-10m-EU-20141129-20141130.png Output from the compare-sigs command examples//N7DR-10m-EU-20141129-20141129.png Output from rank-sig command 73 -- Doc N7DR (N7DR@arrl.net)