-
Notifications
You must be signed in to change notification settings - Fork 3
Configuration Files
Configuration files are the best way to configure the tool. The alternative way of configuring the tool is via command-line arguments. Although powerful, command-line arguments are known to be very verbose, and can consequently, if used, can cause confusion in the future.
Command-line arguments are implemented so that they can serve as overrides. If there are any command line arguments specified, they override any existing configuration specified in the configuration file.
Typical usage can look like:
pyradmon.py --config-file=config.yaml plot
If overrides are needed:
pyradmon.py --config-file=config.yaml plot --plot-define-axes="plot1|sub1|x:ticks=5,label=Hello world!"
Users should always use the configuration file if possible. The configuration file offers a much easier way to configure things, and offers as much power as the command line does. The command line arguments are only to override settings, such as when testing changes to configuration without modifying the configuration file itself.
"OK, OK, I'm convinced. I'm going to write configuration files. But what are they?"
Configuration files have two main components:
- Data sourcing configuration
- Plot generation configuration
The former is a very short configuration defining your source data. The latter is a slightly longer configuration defining the plots to be created.
Both parts of the configuration are stored in one configuration file. Only one configuration file is loaded on execution. The configuration is stored in an easy-to-read format called YAML. That said...
(Yes, that's correct. It's a recursive acronym!)
YAML is a markup language (despite its self-denial). In their own words:
YAML is a human friendly data serialization standard for all programming languages.
YAML is a very clean and organized format, and works perfectly for PyRadmon configuration. Other solutions didn't seem to do well in achieving simple and concise configuration.
- XML: XML is an old but commonly used markup language. For PyRadmon, it has a hierarchical structure and various configuration styles that would prove very useful for us. However, it is very verbose. It also isn't fun to write XML, nor is it very human readable. For data sourcing configuration, this may be fine, but for plot generation configuration, this would be a nightmare to work with.
- JSON: JSON is not as old, and in many places it is also a commonly used markup language. (This is especially true for web applications, as it translates directly to Javascript objects.) For PyRadmon, it has a strong hierarchical structure and various configuration styles and types that would prove very useful for us. However, it is a bit verbose, especially with regards to the brackets and commas required. It isn't as bad as XML to write, but it can be tedious, and it may not look very nice unless it is really spaced out.
- ConfigParser: ConfigParser is a configuration used often in Python, and its style is well known and used everywhere in Windows .ini files. It is a very easy to use and easy to read configuration standard. However, it lacks the hierarchical features that we would need, and it only stores strings for configuration. (To be precise, if you stored "[ 1, 2, 3 ]", it would not be turned into a native type of array(1, 2, 3), and just remain as a string. You could eval() it, but that would be a bad idea...)
And finally:
- YAML: YAML is a new-ish configuration format. It was created around the time JSON was created, but only received attention very recently. It inherits all of the positives of JSON, plus it is easy to read, write, and use! The only catch is that there are some strange aspects to its format (and it's VERY strict about it), but otherwise, it's a very capable format!
For samples to compare with, see footer.
Convinced? Let's learn some YAML!
Let's use an example to work off of:
data:
name: Grocery Store Sales
items: [ pizza, chicken, bacon ]
specials:
- soup
- ice cream
locations:
chicago:
phone_number: 123-456-7890
address: 123 Main St., Chicago, IL USA
inventory: 9000
new_york:
phone_number: 987-654-3210
address: 321 Main St., New York, NY USA
inventory: 1337
slogan: |-
We don't just sell grocery...
...we live and breathe it!The first thing that you might notice is the spacing. It's a very clean spacing, with increased spacing for a certain block of data, and less for others. YAML is organized by spacing - to be specific, each level of data is 2 spaces ahead or backward.
In our example above, our block data has its elements indented with 2 spaces to indicate that those are its own elements. This can be repeated again and again to further levels, therefore forming the hierarchy that we've mentioned previously.
There are 4 basic kinds of data:
- Integers
- Strings
- Lists
- Associative arrays
Integers are simply numbers. For instance, in the example above, 9000 is a number.
Strings are just strings. In YAML, you may specify them with or without quotes. Quotes are useful when you want to force something to be a string. (For instance, specifying '5' instead of 5 makes a string instead of an integer.) In our example, the address is an example of a string. If you want to store a multi-line string, you can use either a pipe and a dash:
key_goes_here: |-
multi value
string goes here... or a string with blank lines representing line breaks, surrounded by quotes:
slogan: "We don't just sell grocery...
...we live and breathe it!"Lists store an array of data. They translate into an array with any kind of data stored inside of it, homogeneous or non-homogeneous. There are two ways to specify lists - with the traditional bracket syntax, or with dashed bullet points. In our example, items is a traditional array, and specials is a bullet point array. Either way will work!
Associative arrays are arrays that store a key and a value. They are useful for storing attributes. In the example above, anything that has a "key" and a "value" is an associative array! (Specifically, you are looking for some text, followed by a colon, and followed by additional text. The entire YAML example is a large example of an associative array, since data is an associative array. Their values can either be indented on a new line, or placed on the same line.
- XML:
<?xml version="1.0" encoding="UTF-8" ?>
<plot1>
<output>plots/test_plot_1_ch%CHANNEL%.png</output>
<plots>
<subplot1_id>
<axes>
<x>
<label />
<ticks>6</ticks>
</x>
<y>
<label />
<ticks>5</ticks>
</y>
</axes>
<data>
<colors>blue</colors>
<colors>red</colors>
<labels>Avg (K)
%AVERAGE%</labels>
<labels>Sdv (K)
%AVERAGE%</labels>
<x>timestamp</x>
<y>ges|bc_total|mean</y>
<y>ges|bc_total|stddev</y>
</data>
<legend>
<border>false</border>
<line>true</line>
</legend>
<title>Total Bias</title>
</subplot1_id>
</plots>
<settings>
<dpi>50</dpi>
<target_size>595</target_size>
<target_size>770</target_size>
</settings>
<title>%INSTRUMENT_SAT% %START_DATE%-%END_DATE%
Channel %CHANNEL% %FREQUENCY% %ASSIMILATION_STATUS%
Global All %EXPERIMENT_ID%</title>
</plot1>- JSON:
{
"plot1": {
"output": "plots/test_plot_1_ch%CHANNEL%.png",
"plots": [
{
"subplot1_id": {
"axes": {
"x": {
"label": null,
"ticks": 6
},
"y": {
"label": null,
"ticks": 5
}
},
"data": {
"colors": [
"blue",
"red"
],
"labels": [
"Avg (K)\n%AVERAGE%",
"Sdv (K)\n%AVERAGE%"
],
"x": "timestamp",
"y": [
"ges|bc_total|mean",
"ges|bc_total|stddev"
]
},
"legend": {
"border": false,
"line": true
},
"title": "Total Bias"
}
}
],
"settings": {
"dpi": 50,
"target_size": [
595,
770
]
},
"title": "%INSTRUMENT_SAT% %START_DATE%-%END_DATE%\nChannel %CHANNEL% %FREQUENCY% %ASSIMILATION_STATUS%\nGlobal All %EXPERIMENT_ID%"
}
}- ConfigParser: Not trivial to write
- YAML:
plot1:
output: plots/test_plot_1_ch%CHANNEL%.png
settings:
dpi: 50
target_size: [595, 770]
title: '%INSTRUMENT_SAT% %START_DATE%-%END_DATE%
Channel %CHANNEL% %FREQUENCY% %ASSIMILATION_STATUS%
Global All %EXPERIMENT_ID%'
plots:
- subplot1_id:
axes:
x: {label: null, ticks: 6}
y: {label: null, ticks: 5}
data:
colors: [blue, red]
labels: ['Avg (K)
%AVERAGE%', 'Sdv (K)
%AVERAGE%']
x: timestamp
y: [ges|bc_total|mean, ges|bc_total|stddev]
legend: {border: false, line: true}
title: Total Bias