@@ -2,6 +2,7 @@ \usepackage{palatino} \usepackage{amsmath} +\usepackage{graphicx} % Idiopidae \usepackage{fancybox} @@ -122,8 +123,100 @@ Due to personal preference, I have included a small library file called \texttt{ \subsection*{Tasks} +The visualizations are organized into Rake tasks. Rake is a standard ruby utility that provides Make-like functionality. These tasks can be run from the command line with \texttt{rake task\_name}. This provides a convenient way to manage the various capabilities of this application. +\subsubsection*{import} +Import loads flight data from \texttt{input/} or the path specified by the \texttt{INPUT\_PATH} variable on the command line. + +Flight data is imported with the function \texttt{Flight::load()}. This function creates a new Flight record, extracting the flight date and flight number from the directory name. It then delegates out to \texttt{DataPoint::from\_files()}. + +\begin{code}{from-files1}{Merging Tuples} +### @include "../models/data_point.rb" "Merge Tuples" +### @end +\end{code} + +From\_files works by generating tuples of the form \texttt{[unix time, ...]} from the ITT data and the insitu data. These tuples are transformed into hashes with the timestamp as the key. This allows them to be easily merged for a given second. Only datapoints with ITT data are of interest, so all tuples with fewer than 10 elements are discarded. + +The DataPoints are then created for the tuples. + +\begin{code}{from-files2}{Creating DataPoints} +### @include "../models/data_point.rb" "Create DataPoints" +### @end +\end{code} + +ITT data is read from the \texttt{*.dbl} files in the \texttt{itt/} directory. Each file is read as a series of records consisting of 9 8-byte double-precisionnumbers in little-endian byte format. These are unpacked into an array. The data\_points tuple is constructed by trimming the dat array. The timestamps are taken from 1904, so the \texttt{EPOCH\_FAIL} (bad pun) constant is subtracted to put them into standard unix time. The data\_points are then averaged to the nearest second from the sample rate of 5Hz. + +\begin{code}{from-itt-data}{From ITT DataPoints} +### @include "../models/data_point.rb" "From ITT DataPoints" +### @end +\end{code} + +The process for reading the GPS data is very similar, except the data are comma delimited. The GPS and ITT data are merged by timestamp into the tuples that are returned. + +The reading of the GPS data and the ITT data take place in parallel, thanks to JRuby's real threads. + + +Insitu data is read from the \texttt{lear*.txt} CSV. The timestamp is in seconds since the midnight of the flight date in EDT (-0400). Fill values are discarded. + +\begin{code}{from-insitu}{From Insitu} +### @include "../models/data_point.rb" "From Insitu" +### @end +\end{code} + +\subsubsection*{plot\_flightpaths} + +Plot\_flightpaths assembles the kml file \texttt{output/datapoint\_paths.kml} which contains the flightpaths of all the flights in the database. The flightpaths are generated from the DataPoint coordinates and represented by a \texttt{KML::LineString}. + +\subsubsection*{plot\_co2\_columns} + +This task creates color coded columns corresponding to the values of the columns of air underneath the flightpath. On top of these columns, boxes are drawn that correspond to the ppm measurement of the insitu instrument at the flight altitude. This visual is output to \texttt{output/co2\_columns.kml}. + +\begin{code}{plot-co2-columns}{Create CO2 Columns} +### @include "../tasks/plot_co2_columns.rake" "Create CO2 Columns" +### @end +\end{code} + +The process for creating the visuals for Google Earth is two-step. First column coordinates are created by calculating the headings between each adjacent datapoint. Points are then found at a radius of 150m that form a line perpendicular to the heading and intersect the first of the datapoints. + +These column coordinates are then handled by \texttt{make\_column\_placemarks}, where the coordinates are used to create squares extruded to the ground. KML style attributes are used in conjunction with the \texttt{CO2ColorCode} to apply color to the columns. Finally, the columns are encapsulated within a placemark for display in Google Earth. + +The InSitu boxes are drawn on top using a similar method. A merging algorithm combines similarly colored adjacent boxes to help reduce file sizes. It seems that this can often reduce the file size by a factor of 2. + +\subsubsection*{make\_color\_bar} + +This task creates a color bar for the CO$_2$ insitu and ITT data. The output is a PNG image at \texttt{output/co2\_color\_bar.png} and a KML file which overlays the image on the screen at \texttt{output/co2\_color\_bar.kml}. The color bar should be the same as Figure (\ref{co2-color-bar}). + +\begin{figure} +\centering +\includegraphics[width=19mm]{../output/co2_color_bar.eps} +\caption{Color Bar for CO$_2$ data.} +\label{co2-color-bar} +\end{figure} + +This is generated using Ruby-Processing in Co2ColorCodeBar and Co2ColorCode classes. The drawing takes place in Co2ColorCodeBar and is pretty standard Processing code. + +The color coding takes place in Co2ColorCode and applies a function to a normalized value (on range [0.0, 1.0]). Insitu and ITT values are normalized to this range through global variables that define the upper and lower bounds for the expected sensor values. + +\begin{code}{color-bounds}{CO2 Bounds Variables} +### @include "../lib/co2_color_code.rb" "Bounds Variables" +### @end +\end{code} + +ITT values are colored through \texttt{Co2ColorCode::itt\_colorify} and Insitu values are colored through \texttt{Co2ColorCode::insitu\_colorify}. Once the values are normalized they are passed to the function \texttt{normalized\_colorify}which assigns colors in the form of $(r,g,b)$ through the function + +\begin{equation} +\label{normalized-colorify} +c(x)= \begin{cases} +(255,255,255), & x < \frac{1}{27} \\ +(-918x+289,-783x+169,255), & x < \frac{6}{27} \\ +(-1215x+448,-1377x+561,255), & x < \frac{11}{27} \\ +(-1296x+719,255,-1296x+719), & x < \frac{16}{27} \\ +(255,-1296x+1023,0), & x < \frac{22}{27} \\ +(-1026x+1052,0,432x-337), & x < \frac{26}{27} \\ +(0,0,0), & x >= \frac{26}{27} \\ +\end{cases} +\end{equation} \end{document} documents/design.tex @@ -1,5 +1,6 @@ require 'lib/ruby-processing' +### @export "Bounds Variables" # global variables for the benefit of Processing # lower and upper bounds for recorded CO2 $itt_low_bound = 0.0 @@ -7,6 +8,7 @@ $itt_high_bound = 5.5 $ins_low_bound = 380 $ins_high_bound = 450 +### @end # Co2ColorCode implements the color scale used to render columns of # CO2 in Google Earth. lib/co2_color_code.rb @@ -3,22 +3,29 @@ EPOCH_FAIL = 2082844800 #seconds between 1904 and 1970 class DataPoint < ActiveRecord::Base belongs_to :flight +### @export "Merge Tuples" def self.from_files(dir_name, flight) - dps = ary_to_time_hash( - from_itt_data(dir_name)).merge( - ary_to_time_hash(from_insitu_data(dir_name, flight.date))) do |k, itt, insitu| - [k] + cdr(itt) + cdr(insitu) - end.values.select{|x| x.size >= 10} + itt_hash = ary_to_time_hash(from_itt_data(dir_name)) + insitu_hash = ary_to_time_hash( + from_insitu_data(dir_name, flight.date)) + + dps = itt_hash.merge(insitu_hash) do |k, itt, insitu| + [k] + cdr(itt) + cdr(insitu) + end.values.select{|x| x.size >= 10} +### @end +### @export "Create DataPoints" dps.each do |dp| DataPoint.create( :time => dp[0].to_i, - :itt_co2 => dp[5]/dp[2], #ratio of backscatter to reference + #ratio of backscatter to reference + :itt_co2 => dp[5]/dp[2], :lat => dp[7], :lon => dp[8], :altitude => dp[9], :insitu_co2 => dp[10], :flight => flight) end +### @end end def to_tuple @@ -34,15 +41,19 @@ class DataPoint < ActiveRecord::Base end private - +### @export "From Insitu" # Array Format: [TimeStamp, CO2_PPM] def self.from_insitu_data(dir_name, date) - co2 = cdr(File.read(Dir.glob("#{dir_name}/insitu/lear*.txt").first).split(/\r?\n/)) + co2 = cdr(File.read(Dir.glob("#{dir_name}/insitu/lear*.txt").first) + co2 = co2.split(/\r?\n/)) + co2.map! do |x| l = x.split(/,\s+/) +### @export "From Insitu" if l[2].to_f != -9999.99 [date.strftime("%s").to_i + l[1].to_i + 4.hours, l[2].to_f] else +### @end nil end end.compact! @@ -55,8 +66,10 @@ class DataPoint < ActiveRecord::Base gps_file = Dir.glob("#{dir_name}/itt/*in-situ_gps_serial_data.txt").first - # Array Format: [TimeStamp,Ref_On,Ref_Side,Ref_Off,Sci_On,Sci_Side,Sci_Off,Lat,Lon,Alt] - + # Array Format: + # [TimeStamp,Ref_On,Ref_Side,Ref_Off,Sci_On,Sci_Side,Sci_Off,Lat,Lon,Alt] + +### @export "From ITT DataPoints" data_points = [] data_points_hash = {} file_thread = Thread.new do @@ -68,7 +81,8 @@ class DataPoint < ActiveRecord::Base dat = IO.read(file, 9*8, offset).unpack(DataFormats::CO2) #adjust for epoch difference - data_points << [dat[0].floor.to_i - EPOCH_FAIL] + dat[1,3] + dat[5,3] + data_points << + [dat[0].floor.to_i - EPOCH_FAIL] + dat[1,3] + dat[5,3] end end @@ -76,6 +90,8 @@ class DataPoint < ActiveRecord::Base data_points_hash = ary_to_time_hash(avg_data_points) end +### @end + gps_hash = {} gps_thread = Thread.new do models/data_point.rb @@ -6,13 +6,14 @@ task :plot_co2_columns do kml = KMLFile.new doc = KML::Document.new(:name => "CO2 Columns") - Flight.find(:all).map do |flight| folder = KML::Folder.new(:name => "Flight #{flight.flight_number}") - + +### @export "Create CO2 Columns" column_coords = make_column_coords(flight.data_points.find(:all)) folder.features = make_column_placemarks(column_coords) +### @end doc.features << folder end @@ -24,7 +25,8 @@ def make_column_coords(dps) heading = 0 distance = 0 column_coords = [] - #assemble datapoint tuples as [Column Pair coordinates, ITT , Insitu, heading] + #assemble datapoint tuples as + #[Column Pair coordinates, ITT , Insitu, heading] dps.each_with_index do |dp, i| if dp.itt_co2 if i < (dps.size-1) @@ -34,11 +36,12 @@ def make_column_coords(dps) distance = KmlTools.great_circle_distance(start, finish) new_heading = KmlTools.heading(start, finish) #angle difference formula - heading = new_heading unless ((((new_heading+180) - heading) % 360) - 180).abs > 90.0 + heading = new_heading unless + ((((new_heading+180) - heading) % 360) - 180).abs > 90.0 end column_coords << [KmlTools.column_pair(dp.lon, dp.lat, dp.altitude, - heading, 150), - dp.itt_co2, dp.insitu_co2, heading] + heading, 150), + dp.itt_co2, dp.insitu_co2, heading] if distance > 200 column_coords << nil end tasks/plot_co2_columns.rake 82800bfa428c67038982b3a44384e64759b847fd Ben Hughes ben@pixelmachine.org http://github.com/schleyfox/ascends_viz/commit/95113e322bd8e6583a9d24e5fbe815c993eaca1d 95113e322bd8e6583a9d24e5fbe815c993eaca1d 2008-07-28T14:35:59-07:00 2008-07-28T14:35:59-07:00 Docos! plus idiopidae markup a587f7caf4d72ee356d010e950c65d2daa01511b Ben Hughes ben@pixelmachine.org