Training on top of an existing model

[ghost]

Hi there,
I am trying to train a new clstm model containing +1000 lines, the training process would take days.
My technique would be to train a couple of hours a day, and continue training the next day, as such.
I created an arabic-8000.clstm model for testing, and added to the script:
load=arabic-8000.clstm
start=8000

But the problem is that clstmocrtrain starts from 0 all over again.
Waiting for your reply

[amitdo]

Please paste here the full commands you used to:

• Train the model
• Load the model

[amitdo]

Which branch/tag are you using? The option to load a model is only supported in the master branch.

[ghost]

@amitdo The train model

set -x
-a
sort -R manifest.txt > /tmp/manifest2.txt
sed 1,100d /tmp/manifest2.txt > train.txt
sed 100q /tmp/manifest2.txt > test.txt

report_every=1000
save_every=1000
maxtrain=50000
target_height=48
dewarp=center
display_every=1000
test_every=1000
hidden=100
lrate=1e-4
save_name=arabic
clstmocrtrain train.txt test.txt

-------------------------------------------------------------
And the Load model is:
set -x
-a
sort -R manifest.txt > /tmp/manifest2.txt
sed 1,100d /tmp/manifest2.txt > train.txt
sed 100q /tmp/manifest2.txt > test.txt

report_every=1000
save_every=1000
maxtrain=50000
target_height=48
dewarp=center
display_every=1000
test_every=1000
hidden=100
lrate=1e-4
save_name=arabic
load=arabic-8000.clstm
start=8000
clstmocrtrain train.txt test.txt

[kba]

@Christophered Your "load"/"train" steps are the same script?

Also, you can enclose multi-line code in triple backticks (```) in markdown, like so:

#!/bin/bash
set -x
-a
sort -R manifest.txt > /tmp/manifest2.txt
sed 1,100d /tmp/manifest2.txt > train.txt
sed 100q /tmp/manifest2.txt > test.txt

report_every=1000
save_every=1000
maxtrain=50000
target_height=48
dewarp=center
display_every=1000
test_every=1000
hidden=100
lrate=1e-4
save_name=arabic
load=arabic-8000.clstm
start=8000
clstmocrtrain train.txt test.txt

[ghost]

@kba loading script is similar to the training script except for the last 3 lines
load=arabic-8000.clstm
start=8000

[ghost]

#129 (comment)
@amitdo what do you mean? I have the default main clstm installed.

[ghost]

How can I train on top of an existing model, or stop and continue training later?

[ghost]

I was using the seperat-derive "legacy" clstm version, it doesn't have save/load options