Monte carlo block added #113
Conversation
…-carlo-block-added
…-carlo-block-added
…-carlo-block-added
…-carlo-block-added
…-carlo-block-added
…-carlo-block-added
…-carlo-block-added
…-carlo-block-added
…-carlo-block-added
| @@ -166,7 +168,19 @@ impl Regressor { | |||
| if mi.ffm_k > 0 { | |||
| let block_ffm = block_ffm::new_ffm_block(&mut bg, mi).unwrap(); | |||
| let triangle_ffm = block_misc::new_triangle_block(&mut bg, block_ffm).unwrap(); | |||
| output = block_misc::new_join_block(&mut bg, vec![output, triangle_ffm]).unwrap(); | |||
| if mi.ffm_mc_iteration_count == 0 || mi.ffm_mc_dropout_rate <= 0.0 { | |||
There was a problem hiding this comment.
Negative dropout rate makes little sense imo?
There was a problem hiding this comment.
Does indeed not make sense, but people make mistakes ;) Thats why I am creating the tiangulation there
| dropout_rate: f32, | ||
| ) -> Result<graph::BlockPtrOutput, Box<dyn Error>> { | ||
| let num_inputs = bg.get_num_output_values(vec![&input]); | ||
| assert_ne!(num_inputs, 0); |
There was a problem hiding this comment.
Why debug, this is when creating? Should break also when running.
There was a problem hiding this comment.
Assertions during runtime cost something - I guess this is fine as it's just during init?
| num_inputs, | ||
| }); | ||
| let mut block_outputs = bg.add_node(block, vec![input])?; | ||
| assert_eq!(block_outputs.len(), 1); |
There was a problem hiding this comment.
Assertions - perhaps explicit to dbg mode
There was a problem hiding this comment.
Why debug, this is when creating? Should break also when running.
There was a problem hiding this comment.
Same answer as above -- as this is just init it's fine, for hotter part of source assertions would imply some kind of perf. penalty
src/block_monte_carlo.rs
Outdated
| } | ||
|
|
||
| fn create_seed_from_input_tape(input_tape: &[f32]) -> u64 { | ||
| (input_tape.iter().sum::<f32>() * 1000.0) as u64 |
There was a problem hiding this comment.
Please add constant to the top of the file + a mini explanation what it is exactly
There was a problem hiding this comment.
Extracted to constant & added explanation. I will keep it close to the actual usage otherwise you constantly need to jump up & down.
There was a problem hiding this comment.
Interesting view on that -- I guess the initial repo's convention is constants on top (if you check older files), even though both approaches are fine
| ) { | ||
| unsafe { | ||
| if self.number_of_inputs_to_skip == 0 { | ||
| self.copy_input_tape_to_output_tape(pb); |
There was a problem hiding this comment.
Afair "direct" implementation of this idea does not require this copy - is there an overhead associated?
There was a problem hiding this comment.
We added an intermediate step, so we still need to copy the input tape to the output tape so it is used in the next step. Its copying values so not that large.
| } | ||
|
|
||
| fn fill_stats(&self, pb: &mut PortBuffer) { | ||
| let mean: f32 = pb.observations.iter().sum::<f32>() / self.num_iterations as f32; |
There was a problem hiding this comment.
What's the overhead of doing this for each prediction? Seems like two pass variance would avoid these extra mults, just a hunch though
or even the Bessel one
There was a problem hiding this comment.
With 5 iterations we get to 0.5 overhead per prediction. So not that much :)
There was a problem hiding this comment.
Right, assuming we stay in that range it's fine then
| } | ||
| let stats_slice = std::slice::from_raw_parts_mut(stats, stats_size); | ||
| *stats_slice.get_unchecked_mut(0) = prediction_stats.mean; | ||
| *stats_slice.get_unchecked_mut(1) = prediction_stats.variance; |
There was a problem hiding this comment.
No need to report std and var, var is enough ..
There was a problem hiding this comment.
We actually need std dev. But I am reporting both just incase we mess up with the calculation.
There was a problem hiding this comment.
Merely saying that stddev is just sqrt(var), computable more or less everywhere (e.g., during analysis)
| if cmd_arguments.is_present("ffm_mc_iteration_count") { | ||
| if let Some(val) = cmd_arguments.value_of("ffm_mc_iteration_count") { | ||
| let hvalue = val.parse::<>()?; | ||
| mi.ffm_mc_iteration_count = hvalue; |
There was a problem hiding this comment.
What exactly is the meaning of the two new hpos?
There was a problem hiding this comment.
ffm_mc_iteration_count - number of iteration that the monte carlo will be run
ffm_mc_dropout_rate - dropout rate for it
Or did I misunderstand?
There was a problem hiding this comment.
Ok cool, just wanted to make sure I understood :) Intuitively mc_iteration_count=1 + mc_dropout_rate=0.0 imply current mode of operation then -- this is not entirely true anymore though right?
Created a separate MonteCarlo block which executes the run of a specific continuation of blocks
ntimes (wherenis the number of iterations), where a specific number of inputs (also configurable) is masked with a random function where the seed is based on the sum of the input. For each run we pick which index should be masked instead of iterating over the whole array & checking if it should be masked or not.In the case of FW this block can be configurable run as part of the FFM after the triangle block, where it re-uses the input from the FFM/triangle on each run, which reduces the CPU & time of execution.
The execution of the sub runs is stored into a separate
PredictionStatsstruct, which contains the mean, standard deviation, variance and number of iterations.Additionally we allow the exposing of the
PredictionStatsinformation via FFI as an array of outputs.