New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GSoC] Added LSTM baseline model with copy task benchmark #1

Open
wants to merge 18 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@17minutes

17minutes commented Jun 23, 2017

This PR is part of my GSoC project "Augmented RNNs".
Implemented a basic LSTM model for AddTask from the running version of my mlpack fork.

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jun 23, 2017

By the way, how can I suppress SGD output in log and at the same time see my [DEBUG] output?

17minutes commented Jun 23, 2017

By the way, how can I suppress SGD output in log and at the same time see my [DEBUG] output?

@17minutes 17minutes changed the title from Added LSTM baseline model with copy task benchmark to [GSoC] Added LSTM baseline model with copy task benchmark Jun 23, 2017

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jun 23, 2017

Member

Thanks for the PR. Regarding the logging you could set:

mlpack::Log::Debug.ignoreInput = false;
mlpack::Log::Info.ignoreInput = true;
mlpack::Log::Warn.ignoreInput = true;
Member

zoq commented Jun 23, 2017

Thanks for the PR. Regarding the logging you could set:

mlpack::Log::Debug.ignoreInput = false;
mlpack::Log::Info.ignoreInput = true;
mlpack::Log::Warn.ignoreInput = true;
@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jun 23, 2017

Can't compile it:

/home/pretorium/Experiments/mlpack-models/models/augmented_lstm_baseline/src/lstm_baseline_main.cpp:276:14: error: ‘class mlpack::util::NullOutStream’ has no member named ‘ignoreInput’
   Log::Debug.ignoreInput = false;

17minutes commented Jun 23, 2017

Can't compile it:

/home/pretorium/Experiments/mlpack-models/models/augmented_lstm_baseline/src/lstm_baseline_main.cpp:276:14: error: ‘class mlpack::util::NullOutStream’ has no member named ‘ignoreInput’
   Log::Debug.ignoreInput = false;
@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jun 23, 2017

Member

Right this only works if you build with DEBUG=ON, but it looks like you build with DEBUG=OFF?

Member

zoq commented Jun 23, 2017

Right this only works if you build with DEBUG=ON, but it looks like you build with DEBUG=OFF?

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jun 29, 2017

About logging: with DEBUG=ON this didn't work either.

About the last commit: trying to make an AddTask solver, but something goes wrong in a very weird way. I have already found an error (the sequences must be aligned by length), fixing.

17minutes commented Jun 29, 2017

About logging: with DEBUG=ON this didn't work either.

About the last commit: trying to make an AddTask solver, but something goes wrong in a very weird way. I have already found an error (the sequences must be aligned by length), fixing.

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jun 29, 2017

Well, I fixed it faster than I thought 😄

Speaking about the LSTM model itself, it performs real poor on AddTask - it scored ~14.5% on the task of adding 2-bit numbers.

17minutes commented Jun 29, 2017

Well, I fixed it faster than I thought 😄

Speaking about the LSTM model itself, it performs real poor on AddTask - it scored ~14.5% on the task of adding 2-bit numbers.

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jun 29, 2017

I think I'll also create a a TODO list similar to the one from main mlpack repo:

  • Create a function for creating a model (so far we have a copy-paste code in all three methods);
  • Shuffle the examples in the train run;
  • Evaluate the results and write them to the report.

17minutes commented Jun 29, 2017

I think I'll also create a a TODO list similar to the one from main mlpack repo:

  • Create a function for creating a model (so far we have a copy-paste code in all three methods);
  • Shuffle the examples in the train run;
  • Evaluate the results and write them to the report.
@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jun 29, 2017

Member

I'll take a look at the AddTask results, in my tested for the Copy Task, I usually use SGD or Mini Batch SGD to achieve better results. Can you push the Add and Sort task to the mlpack repo?

Member

zoq commented Jun 29, 2017

I'll take a look at the AddTask results, in my tested for the Copy Task, I usually use SGD or Mini Batch SGD to achieve better results. Can you push the Add and Sort task to the mlpack repo?

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jun 30, 2017

Can you push the Add and Sort task to the mlpack repo?

Sorry, didn't quite understand that - those already seem to be in partobs-mdp/mlpack. Can you restate that, please?

17minutes commented Jun 30, 2017

Can you push the Add and Sort task to the mlpack repo?

Sorry, didn't quite understand that - those already seem to be in partobs-mdp/mlpack. Can you restate that, please?

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jun 30, 2017

Member

Thanks for the clarification, I guess relaying on a binary representation might be helpful if you are going to normalise the data later. So I think the next step is to transform the output into the representation as done for the copy task? But instead of two inputs we use 3 inputs?

Yes, that's precisely what I'm trying to do now. The transform shouldn't be too hard - inputs.reshape, maybe preceeded with inputs.t()

I can't see that you changed the representation of the Sort or Add task, maybe I missed something?

Member

zoq commented Jun 30, 2017

Thanks for the clarification, I guess relaying on a binary representation might be helpful if you are going to normalise the data later. So I think the next step is to transform the output into the representation as done for the copy task? But instead of two inputs we use 3 inputs?

Yes, that's precisely what I'm trying to do now. The transform shouldn't be too hard - inputs.reshape, maybe preceeded with inputs.t()

I can't see that you changed the representation of the Sort or Add task, maybe I missed something?

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jun 30, 2017

Member

Ahh I see, you did the reshape inside the main file, I think it's a good idea to do this inside the task itself. What do you think?

Member

zoq commented Jun 30, 2017

Ahh I see, you did the reshape inside the main file, I think it's a good idea to do this inside the task itself. What do you think?

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jun 30, 2017

Ahh I see, you did the reshape inside the main file, I think it's a good idea to do this inside the task itself. What do you think?

Sure, doing it right now.

17minutes commented Jun 30, 2017

Ahh I see, you did the reshape inside the main file, I think it's a good idea to do this inside the task itself. What do you think?

Sure, doing it right now.

@rcurtin

Looks like this is still an active WIP so I just added some general comments. Glad to see the models repo useful. :) We don't have any style or build checker here so I guess you will have to do that yourself until we get that set up.

@rcurtin

Looks good to me, I tested it out on my system. I got good performance with the copy task, but the add task seemed to have problems:

$ bin/mlpack_lstm_baseline -t add -i 1 -b 3 -r 1 -s 128 -v
...
Final score: 0.117188

The same with the sort task:

$ bin/mlpack_lstm_baseline -t sort -i 1 -l 5 -b 3 -r 1 -s 128 -v
...
[INFO ] Final score: 0.0234375

Did I run any of those simulations wrong, or is this the expected performance? I built against your mlpack master branch so I think I did everything right...

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jul 11, 2017

Did I run any of those simulations wrong, or is this the expected performance?

Everything goes as planned :) Neither me nor @zoq (as far as I remember) didn't manage to get outstanding performance of LSTM model on those two tasks.

17minutes commented Jul 11, 2017

Did I run any of those simulations wrong, or is this the expected performance?

Everything goes as planned :) Neither me nor @zoq (as far as I remember) didn't manage to get outstanding performance of LSTM model on those two tasks.

@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Jul 11, 2017

Member

Ok, good to hear I did not encounter any issues that weren't expected. If the performance is not so good, I wonder if the fix about the epochs I mentioned will allow it to train better---like if we use a very large number of epochs or something, as opposed to the current limit of 100k SGD iterations. If you fix the code, I am happy to fire-and-forget run it in the background and let you know what the results are in a few days. :)

Member

rcurtin commented Jul 11, 2017

Ok, good to hear I did not encounter any issues that weren't expected. If the performance is not so good, I wonder if the fix about the epochs I mentioned will allow it to train better---like if we use a very large number of epochs or something, as opposed to the current limit of 100k SGD iterations. If you fix the code, I am happy to fire-and-forget run it in the background and let you know what the results are in a few days. :)

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jul 12, 2017

Did I wrap the line correctly?

17minutes commented Jul 12, 2017

Did I wrap the line correctly?

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jul 12, 2017

Member

Did I wrap the line correctly?

Yes, looks good. I have to admit that we are really picky about style issues, sorry.

Member

zoq commented Jul 12, 2017

Did I wrap the line correctly?

Yes, looks good. I have to admit that we are really picky about style issues, sorry.

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Jul 12, 2017

I have to admit that we are really picky about style issues, sorry.

I understand, no problem - otherwise the codebase would already degenerate into something completely unmaintainable and downright dirty :)

17minutes commented Jul 12, 2017

I have to admit that we are really picky about style issues, sorry.

I understand, no problem - otherwise the codebase would already degenerate into something completely unmaintainable and downright dirty :)

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jul 12, 2017

Member

Here are the updated results:

➜  bin git:(master) ✗ ./run_sort_task.sh 
Evaluating length = 1, binary_length = 2
Evaluating length = 2, binary_length = 2
Final score: 0.578125
Evaluating length = 3, binary_length = 2
Final score: 0.273438
Evaluating length = 4, binary_length = 2
Final score: 0.226562
Evaluating length = 5, binary_length = 2
Final score: 0.1875
Evaluating length = 6, binary_length = 2
Final score: 0.0859375
Evaluating length = 7, binary_length = 2
Final score: 0.046875
Evaluating length = 8, binary_length = 2
Final score: 0.0390625
Evaluating length = 1, binary_length = 3
Evaluating length = 2, binary_length = 3
Final score: 0.445312
Evaluating length = 3, binary_length = 3
Final score: 0.132812
Evaluating length = 4, binary_length = 3
Final score: 0.015625
Evaluating length = 5, binary_length = 3
Final score: 0.0078125
Evaluating length = 1, binary_length = 4
Evaluating length = 2, binary_length = 4
Final score: 0.320312
Evaluating length = 3, binary_length = 4
Final score: 0.03125
Evaluating length = 4, binary_length = 4
Final score: 0.0234375
Evaluating length = 1, binary_length = 5
Evaluating length = 2, binary_length = 5
Final score: 0.21875
Evaluating length = 3, binary_length = 5
Final score: 0
Evaluating length = 1, binary_length = 6
Evaluating length = 2, binary_length = 6
Final score: 0.0546875
Evaluating length = 1, binary_length = 7
Evaluating length = 2, binary_length = 7
Final score: 0.03125
Evaluating length = 1, binary_length = 8
Evaluating length = 2, binary_length = 8
Final score: 0.0234375
Evaluating length = 1, binary_length = 9
Evaluating length = 1, binary_length = 10
➜  bin git:(master) ✗ ./run_copy_task.sh  
Evaluating length = 2
Final score: 1
Evaluating length = 3
Final score: 1
Evaluating length = 4
Final score: 1
Evaluating length = 5
Final score: 1
Evaluating length = 6
Final score: 1
Evaluating length = 7
Final score: 0.96875
Evaluating length = 8
Final score: 0.421875
Evaluating length = 9
Final score: 0.84375
Evaluating length = 10
Final score: 0.820312
➜  bin git:(master) ✗ ./run_add_task.sh 
Evaluating bit_length 1
Final score: 0.460938
Evaluating bit_length 2
Final score: 0.28125
Evaluating bit_length 3
Final score: 0.109375
Evaluating bit_length 4
Final score: 0.0390625
Evaluating bit_length 5
Final score: 0.0234375
Evaluating bit_length 6
Final score: 0.0234375
Evaluating bit_length 7
Final score: 0
Evaluating bit_length 8
Final score: 0.0078125
Evaluating bit_length 9
Final score: 0
Evaluating bit_length 10
Final score: 0
Member

zoq commented Jul 12, 2017

Here are the updated results:

➜  bin git:(master) ✗ ./run_sort_task.sh 
Evaluating length = 1, binary_length = 2
Evaluating length = 2, binary_length = 2
Final score: 0.578125
Evaluating length = 3, binary_length = 2
Final score: 0.273438
Evaluating length = 4, binary_length = 2
Final score: 0.226562
Evaluating length = 5, binary_length = 2
Final score: 0.1875
Evaluating length = 6, binary_length = 2
Final score: 0.0859375
Evaluating length = 7, binary_length = 2
Final score: 0.046875
Evaluating length = 8, binary_length = 2
Final score: 0.0390625
Evaluating length = 1, binary_length = 3
Evaluating length = 2, binary_length = 3
Final score: 0.445312
Evaluating length = 3, binary_length = 3
Final score: 0.132812
Evaluating length = 4, binary_length = 3
Final score: 0.015625
Evaluating length = 5, binary_length = 3
Final score: 0.0078125
Evaluating length = 1, binary_length = 4
Evaluating length = 2, binary_length = 4
Final score: 0.320312
Evaluating length = 3, binary_length = 4
Final score: 0.03125
Evaluating length = 4, binary_length = 4
Final score: 0.0234375
Evaluating length = 1, binary_length = 5
Evaluating length = 2, binary_length = 5
Final score: 0.21875
Evaluating length = 3, binary_length = 5
Final score: 0
Evaluating length = 1, binary_length = 6
Evaluating length = 2, binary_length = 6
Final score: 0.0546875
Evaluating length = 1, binary_length = 7
Evaluating length = 2, binary_length = 7
Final score: 0.03125
Evaluating length = 1, binary_length = 8
Evaluating length = 2, binary_length = 8
Final score: 0.0234375
Evaluating length = 1, binary_length = 9
Evaluating length = 1, binary_length = 10
➜  bin git:(master) ✗ ./run_copy_task.sh  
Evaluating length = 2
Final score: 1
Evaluating length = 3
Final score: 1
Evaluating length = 4
Final score: 1
Evaluating length = 5
Final score: 1
Evaluating length = 6
Final score: 1
Evaluating length = 7
Final score: 0.96875
Evaluating length = 8
Final score: 0.421875
Evaluating length = 9
Final score: 0.84375
Evaluating length = 10
Final score: 0.820312
➜  bin git:(master) ✗ ./run_add_task.sh 
Evaluating bit_length 1
Final score: 0.460938
Evaluating bit_length 2
Final score: 0.28125
Evaluating bit_length 3
Final score: 0.109375
Evaluating bit_length 4
Final score: 0.0390625
Evaluating bit_length 5
Final score: 0.0234375
Evaluating bit_length 6
Final score: 0.0234375
Evaluating bit_length 7
Final score: 0
Evaluating bit_length 8
Final score: 0.0078125
Evaluating bit_length 9
Final score: 0
Evaluating bit_length 10
Final score: 0
@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Jul 13, 2017

Member

To add on to the discussion about training:

The problem I see is, that we train MaxIterations times on a single sample, since right now there is no interface for arma::field; it should be easy to implement such an interface, or we could just use the fixed sequence representation and use arma::mat as input.

Using arma::field the way we do it right now introduces another problem, at the beginning of each Optimize(...) call, we instantiate a new Adam update policy, which should be used over iterations.

As the code stands right now, it is using the fixed-length representation, but the testing is done on the non-fixed-length representation. I would suspect that this might cause some decrease in performance, because the model always "expects" the sequence to be the same length as its training sequences (I am handwaving but that hopefully kind of makes sense). I think maybe there are also some other issues, noted in mlpack#1005 about the representation of the sequence, but I do think this may play into it a little bit.

But, currently there is no way to pass non-fixed-length data. I seem to remember having this discussion in another issue or PR, but I can't find it now. @zoq, what do you think, should the RNN::Train() interface be changed to accept arma::field<arma::mat>, where each arma::mat is a sequence to train on? I can't think of any other good solution to accept a set of non-fixed-length sequence inputs. (That would then mean we should change the HMM interface, which uses std::vector<arma::mat> to accept a bunch of sequences, but of course that is no problem.)

Member

rcurtin commented Jul 13, 2017

To add on to the discussion about training:

The problem I see is, that we train MaxIterations times on a single sample, since right now there is no interface for arma::field; it should be easy to implement such an interface, or we could just use the fixed sequence representation and use arma::mat as input.

Using arma::field the way we do it right now introduces another problem, at the beginning of each Optimize(...) call, we instantiate a new Adam update policy, which should be used over iterations.

As the code stands right now, it is using the fixed-length representation, but the testing is done on the non-fixed-length representation. I would suspect that this might cause some decrease in performance, because the model always "expects" the sequence to be the same length as its training sequences (I am handwaving but that hopefully kind of makes sense). I think maybe there are also some other issues, noted in mlpack#1005 about the representation of the sequence, but I do think this may play into it a little bit.

But, currently there is no way to pass non-fixed-length data. I seem to remember having this discussion in another issue or PR, but I can't find it now. @zoq, what do you think, should the RNN::Train() interface be changed to accept arma::field<arma::mat>, where each arma::mat is a sequence to train on? I can't think of any other good solution to accept a set of non-fixed-length sequence inputs. (That would then mean we should change the HMM interface, which uses std::vector<arma::mat> to accept a bunch of sequences, but of course that is no problem.)

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Jul 20, 2017

Member

But, currently there is no way to pass non-fixed-length data. I seem to remember having this discussion in another issue or PR, but I can't find it now. @zoq, what do you think, should the RNN::Train() interface be changed to accept arma::fieldarma::mat, where each arma::mat is a sequence to train on? I can't think of any other good solution to accept a set of non-fixed-length sequence inputs. (That would then mean we should change the HMM interface, which uses std::vectorarma::mat to accept a bunch of sequences, but of course that is no problem.)

I think we should provide a Train interface for arma::mat and arma::field, in case someone like sto use the same data as for training e.g. a feedforward network, we don't have to transform the data into arma::field. Also, I don't mind to use std::vector<arma::mat>instead of arma::filed, however for me arma:: field feels more natural, and I think has some additional benefits like .n_elem.

Member

zoq commented Jul 20, 2017

But, currently there is no way to pass non-fixed-length data. I seem to remember having this discussion in another issue or PR, but I can't find it now. @zoq, what do you think, should the RNN::Train() interface be changed to accept arma::fieldarma::mat, where each arma::mat is a sequence to train on? I can't think of any other good solution to accept a set of non-fixed-length sequence inputs. (That would then mean we should change the HMM interface, which uses std::vectorarma::mat to accept a bunch of sequences, but of course that is no problem.)

I think we should provide a Train interface for arma::mat and arma::field, in case someone like sto use the same data as for training e.g. a feedforward network, we don't have to transform the data into arma::field. Also, I don't mind to use std::vector<arma::mat>instead of arma::filed, however for me arma:: field feels more natural, and I think has some additional benefits like .n_elem.

@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Jul 20, 2017

Member

Agreed---arma::mat can assume a fixed-length representation, and arma::field can assume a non-fixed length representation. I also agree that arma::field<arma::mat> is more natural than std::vector<arma::mat>.

Member

rcurtin commented Jul 20, 2017

Agreed---arma::mat can assume a fixed-length representation, and arma::field can assume a non-fixed length representation. I also agree that arma::field<arma::mat> is more natural than std::vector<arma::mat>.

@sumedhghaisas

This comment has been minimized.

Show comment
Hide comment
@sumedhghaisas

sumedhghaisas Aug 3, 2017

Member

I ran some experiments and found out that RMSProp performs the best. The learning rate I used was 0.0001 with default momentum. The learning for LSTM was much more stable and much faster. Each iteration with RMSProp is 3 times faster than that of Minibatch SGD (on my machine). Same for NTM.

Member

sumedhghaisas commented Aug 3, 2017

I ran some experiments and found out that RMSProp performs the best. The learning rate I used was 0.0001 with default momentum. The learning for LSTM was much more stable and much faster. Each iteration with RMSProp is 3 times faster than that of Minibatch SGD (on my machine). Same for NTM.

@rcurtin

This comment has been minimized.

Show comment
Hide comment
@rcurtin

rcurtin Aug 6, 2017

Member

Sumedh, thanks for running the experiments. I think that once RMSprop and the other optimizers support minibatch learning, there will be some additional improvement. Do you have the performance numbers for your simulations?

@partobs-mdp: what would you say the status of this is? I realize I may have missed some IRC discussions while I was traveling. Does this need to be further reviewed, or is it waiting on some more implementation from your end, or is it ready to merge?

Member

rcurtin commented Aug 6, 2017

Sumedh, thanks for running the experiments. I think that once RMSprop and the other optimizers support minibatch learning, there will be some additional improvement. Do you have the performance numbers for your simulations?

@partobs-mdp: what would you say the status of this is? I realize I may have missed some IRC discussions while I was traveling. Does this need to be further reviewed, or is it waiting on some more implementation from your end, or is it ready to merge?

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Aug 6, 2017

@rcurtin: It's mostly finished, but @zoq (if I remember correctly) has promised to run LSTM model on "shifted" representation of AddTask and SortTask and show the results.

17minutes commented Aug 6, 2017

@rcurtin: It's mostly finished, but @zoq (if I remember correctly) has promised to run LSTM model on "shifted" representation of AddTask and SortTask and show the results.

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Aug 6, 2017

Member

First results with the shifted version:

➜  bin git:(master) ✗ ./run_sort_task.sh 
Evaluating length = 1, binary_length = 2
Evaluating length = 2, binary_length = 2
Average score: 1
Evaluating length = 3, binary_length = 2
Average score: 1
Evaluating length = 4, binary_length = 2
Average score: 0.851562
Evaluating length = 5, binary_length = 2
Average score: 0.929688
Evaluating length = 6, binary_length = 2
Average score: 0.648438
Evaluating length = 7, binary_length = 2
Average score: 0.671875
Evaluating length = 8, binary_length = 2
Average score: 0.671875
Evaluating length = 1, binary_length = 3
Evaluating length = 2, binary_length = 3
Average score: 0.945312
Evaluating length = 3, binary_length = 3
Average score: 0.601562
Evaluating length = 4, binary_length = 3
Average score: 0.234375
Evaluating length = 5, binary_length = 3
Average score: 0.148438
Evaluating length = 1, binary_length = 4
Evaluating length = 2, binary_length = 4
Average score: 0.710938
Evaluating length = 3, binary_length = 4
Average score: 0.0703125
Evaluating length = 4, binary_length = 4
Average score: 0
Evaluating length = 1, binary_length = 5
Evaluating length = 2, binary_length = 5
Average score: 0.21875
Evaluating length = 3, binary_length = 5
Average score: 0.015625
Evaluating length = 1, binary_length = 6
Evaluating length = 2, binary_length = 6
Average score: 0.078125
Evaluating length = 1, binary_length = 7
Evaluating length = 2, binary_length = 7
Average score: 0
Evaluating length = 1, binary_length = 8
Evaluating length = 2, binary_length = 8
Average score: 0.0078125
Evaluating length = 1, binary_length = 9
Evaluating length = 1, binary_length = 10

I think if we adjust some network parameter we can improve the results.

Member

zoq commented Aug 6, 2017

First results with the shifted version:

➜  bin git:(master) ✗ ./run_sort_task.sh 
Evaluating length = 1, binary_length = 2
Evaluating length = 2, binary_length = 2
Average score: 1
Evaluating length = 3, binary_length = 2
Average score: 1
Evaluating length = 4, binary_length = 2
Average score: 0.851562
Evaluating length = 5, binary_length = 2
Average score: 0.929688
Evaluating length = 6, binary_length = 2
Average score: 0.648438
Evaluating length = 7, binary_length = 2
Average score: 0.671875
Evaluating length = 8, binary_length = 2
Average score: 0.671875
Evaluating length = 1, binary_length = 3
Evaluating length = 2, binary_length = 3
Average score: 0.945312
Evaluating length = 3, binary_length = 3
Average score: 0.601562
Evaluating length = 4, binary_length = 3
Average score: 0.234375
Evaluating length = 5, binary_length = 3
Average score: 0.148438
Evaluating length = 1, binary_length = 4
Evaluating length = 2, binary_length = 4
Average score: 0.710938
Evaluating length = 3, binary_length = 4
Average score: 0.0703125
Evaluating length = 4, binary_length = 4
Average score: 0
Evaluating length = 1, binary_length = 5
Evaluating length = 2, binary_length = 5
Average score: 0.21875
Evaluating length = 3, binary_length = 5
Average score: 0.015625
Evaluating length = 1, binary_length = 6
Evaluating length = 2, binary_length = 6
Average score: 0.078125
Evaluating length = 1, binary_length = 7
Evaluating length = 2, binary_length = 7
Average score: 0
Evaluating length = 1, binary_length = 8
Evaluating length = 2, binary_length = 8
Average score: 0.0078125
Evaluating length = 1, binary_length = 9
Evaluating length = 1, binary_length = 10

I think if we adjust some network parameter we can improve the results.

@17minutes

This comment has been minimized.

Show comment
Hide comment
@17minutes

17minutes Aug 7, 2017

@zoq: Imho we don't have to adjust the model - our goal was to prove our task class and model are not broken.

17minutes commented Aug 7, 2017

@zoq: Imho we don't have to adjust the model - our goal was to prove our task class and model are not broken.

@zoq

This comment has been minimized.

Show comment
Hide comment
@zoq

zoq Aug 7, 2017

Member

For completeness:

➜  bin git:(master) ✗ ./run_add_task.sh
Evaluating bit_length 1
Average score: 1
Evaluating bit_length 2
Average score: 1
Evaluating bit_length 3
Average score: 0.96875
Evaluating bit_length 4
Average score: 0.484375
Evaluating bit_length 5
Average score: 0.0078125
Evaluating bit_length 6
Average score: 0.015625
Evaluating bit_length 7
Average score: 0.0078125
Evaluating bit_length 8
Average score: 0
Evaluating bit_length 9
Average score: 0
Evaluating bit_length 10
Average score: 0

I agree, searching for optimal parameter isn't part of this project, I think mlpack/mlpack#1005 is ready to be merged.

Member

zoq commented Aug 7, 2017

For completeness:

➜  bin git:(master) ✗ ./run_add_task.sh
Evaluating bit_length 1
Average score: 1
Evaluating bit_length 2
Average score: 1
Evaluating bit_length 3
Average score: 0.96875
Evaluating bit_length 4
Average score: 0.484375
Evaluating bit_length 5
Average score: 0.0078125
Evaluating bit_length 6
Average score: 0.015625
Evaluating bit_length 7
Average score: 0.0078125
Evaluating bit_length 8
Average score: 0
Evaluating bit_length 9
Average score: 0
Evaluating bit_length 10
Average score: 0

I agree, searching for optimal parameter isn't part of this project, I think mlpack/mlpack#1005 is ready to be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment