Problem with load checkpoint #149
Comments
|
Checkpoint file should be in the ckpt folder, move it back . It should work |
|
move files in ckpt/cfg to ckpt/checkpoint and try it |
|
Thank you It's work. |
|
I met the same problem as you,could you show me your entire command? |
|
@beebrain can you share the full command which worked for you? |
|
step 124 - loss 60.73698425292969 - moving ave loss 64.46739123826117 For me, it's working till 125 epochs and then this error is coming. |
|
@anushabhura This can be probably your computer ran out of storage and it cannot create a checkpoint at 125th step. You can use Google co-lab with your google drive. |
|
Thanks for your reply but actually I am using anaconda jupyter notebook.
Can we do something about it?
…On Thu 6 Feb, 2020, 12:39 AM K.K.D.A.K.Indrajith, ***@***.***> wrote:
@anushabhura <https://github.com/anushabhura> This can be probably your
computer ran out of storage and it cannot create a checkpoint at 125th
step. You can use Google co-lab with your google drive.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#149?email_source=notifications&email_token=AKUL53NE6WNCIHECN6PG2D3RBMFHLA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK4TNMI#issuecomment-582563505>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKUL53L6OC24H7MUHYXU5JLRBMFHLANCNFSM4DG6R6HQ>
.
|
|
#869
I tried this too but this command (python flow --model
cfg/tiny-yolo-voc-1c.cfg --load -1 --savepb) is giving me error.
Traceback (most recent call last):
File "flow", line 6, in <module>
cliHandler(sys.argv)
File "C:\Users\intel\darkflow\cli.py", line 26, in cliHandler
tfnet = TFNet(FLAGS)
File "C:\Users\intel\darkflow\net\build.py", line 88, in __init__
self.setup_meta_ops()
File "C:\Users\intel\darkflow\net\build.py", line 163, in setup_meta_ops
if self.FLAGS.load != 0: self.load_from_ckpt()
File "C:\Users\intel\darkflow\net\help.py", line 23, in load_from_ckpt
with open(self.FLAGS.backup + 'checkpoint', 'r') as f:
PermissionError: [Errno 13] Permission denied: './ckpt/checkpoint'
In anaconda jupyter, ckpt folder has been created by program but is not
creating any file inside ckpt.
…On Thu, Feb 6, 2020 at 12:39 AM K.K.D.A.K.Indrajith < ***@***.***> wrote:
@anushabhura <https://github.com/anushabhura> This can be probably your
computer ran out of storage and it cannot create a checkpoint at 125th
step. You can use Google co-lab with your google drive.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#149?email_source=notifications&email_token=AKUL53NE6WNCIHECN6PG2D3RBMFHLA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK4TNMI#issuecomment-582563505>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKUL53L6OC24H7MUHYXU5JLRBMFHLANCNFSM4DG6R6HQ>
.
|
Could you show an example command when you executed your code? |
|
python flow --model cfg/tiny-yolo-voc-1c.cfg --load tiny-yolo-voc-.weights
--train --annotation darkflow/annotation --dataset
darkflow/image_files --epoch 250
I am using above command and its again giving me this checkpoint error .
Finish 97 epoch(es)
step 98 - loss 106.16637420654297 - moving ave loss 106.27569753284817
Finish 98 epoch(es)
step 99 - loss 106.19925689697266 - moving ave loss 106.26805346926062
Finish 99 epoch(es)
step 100 - loss 106.1893081665039 - moving ave loss 106.26017893898495
Traceback (most recent call last):
File "flow", line 6, in <module>
cliHandler(sys.argv)
File "C:\Users\intel\darkflow\cli.py", line 33, in cliHandler
print('Enter training ...'); tfnet.train()
File "C:\Users\intel\darkflow\net\flow.py", line 66, in train
if not ckpt: _save_ckpt(self, *args)
File "C:\Users\intel\darkflow\net\flow.py", line 21, in _save_ckpt
with open(profile, 'wb') as profile_ckpt:
FileNotFoundError: [Errno 2] No such file or directory:
'./ckpt/cfg/tiny-yolo-vo
c-1c-100.profile'
On Thu, Feb 6, 2020 at 10:14 AM Pisit Nakjai <notifications@github.com>
wrote:
… #869 <#869> I tried this too
but this command (python flow --model cfg/tiny-yolo-voc-1c.cfg --load -1
--savepb) is giving me error. Traceback (most recent call last): File
"flow", line 6, in cliHandler(sys.argv) File
"C:\Users\intel\darkflow\cli.py", line 26, in cliHandler tfnet =
TFNet(FLAGS) File "C:\Users\intel\darkflow\net\build.py", line 88, in
*init* self.setup_meta_ops() File "C:\Users\intel\darkflow\net\build.py",
line 163, in setup_meta_ops if self.FLAGS.load != 0: self.load_from_ckpt()
File "C:\Users\intel\darkflow\net\help.py", line 23, in load_from_ckpt with
open(self.FLAGS.backup + 'checkpoint', 'r') as f: PermissionError: [Errno
13] Permission denied: './ckpt/checkpoint' In anaconda jupyter, ckpt folder
has been created by program but is not creating any file inside ckpt.
… <#m_5662571605862279490_>
On Thu, Feb 6, 2020 at 12:39 AM K.K.D.A.K.Indrajith < *@*.***> wrote:
@anushabhura <https://github.com/anushabhura>
https://github.com/anushabhura This can be probably your computer ran out
of storage and it cannot create a checkpoint at 125th step. You can use
Google co-lab with your google drive. — You are receiving this because you
were mentioned. Reply to this email directly, view it on GitHub <#149
<#149>?email_source=notifications&email_token=AKUL53NE6WNCIHECN6PG2D3RBMFHLA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK4TNMI#issuecomment-582563505>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKUL53L6OC24H7MUHYXU5JLRBMFHLANCNFSM4DG6R6HQ
.
Could you show an example command when you executed your code?
I think, your script need permission to write the checkpoint file.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#149?email_source=notifications&email_token=AKUL53ISUS62J4ZYEWMWKTTRBOIURA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK55G3I#issuecomment-582734701>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKUL53OHS34AJH4EKRPSKDTRBOIURANCNFSM4DG6R6HQ>
.
|
|
I think this file tiny-yolo-voc-1c-100.profile should be created but my
ckpt folder is empty every time i do training .
…On Thu, Feb 6, 2020 at 10:21 AM Anusha Bhura ***@***.***> wrote:
python flow --model cfg/tiny-yolo-voc-1c.cfg --load tiny-yolo-voc-.weights
--train --annotation darkflow/annotation --dataset
darkflow/image_files --epoch 250
I am using above command and its again giving me this checkpoint error .
Finish 97 epoch(es)
step 98 - loss 106.16637420654297 - moving ave loss 106.27569753284817
Finish 98 epoch(es)
step 99 - loss 106.19925689697266 - moving ave loss 106.26805346926062
Finish 99 epoch(es)
step 100 - loss 106.1893081665039 - moving ave loss 106.26017893898495
Traceback (most recent call last):
File "flow", line 6, in <module>
cliHandler(sys.argv)
File "C:\Users\intel\darkflow\cli.py", line 33, in cliHandler
print('Enter training ...'); tfnet.train()
File "C:\Users\intel\darkflow\net\flow.py", line 66, in train
if not ckpt: _save_ckpt(self, *args)
File "C:\Users\intel\darkflow\net\flow.py", line 21, in _save_ckpt
with open(profile, 'wb') as profile_ckpt:
FileNotFoundError: [Errno 2] No such file or directory:
'./ckpt/cfg/tiny-yolo-vo
c-1c-100.profile'
On Thu, Feb 6, 2020 at 10:14 AM Pisit Nakjai ***@***.***>
wrote:
> #869 <#869> I tried this too
> but this command (python flow --model cfg/tiny-yolo-voc-1c.cfg --load -1
> --savepb) is giving me error. Traceback (most recent call last): File
> "flow", line 6, in cliHandler(sys.argv) File
> "C:\Users\intel\darkflow\cli.py", line 26, in cliHandler tfnet =
> TFNet(FLAGS) File "C:\Users\intel\darkflow\net\build.py", line 88, in
> *init* self.setup_meta_ops() File
> "C:\Users\intel\darkflow\net\build.py", line 163, in setup_meta_ops if
> self.FLAGS.load != 0: self.load_from_ckpt() File
> "C:\Users\intel\darkflow\net\help.py", line 23, in load_from_ckpt with
> open(self.FLAGS.backup + 'checkpoint', 'r') as f: PermissionError: [Errno
> 13] Permission denied: './ckpt/checkpoint' In anaconda jupyter, ckpt folder
> has been created by program but is not creating any file inside ckpt.
> … <#m_3992249403181806711_m_5662571605862279490_>
> On Thu, Feb 6, 2020 at 12:39 AM K.K.D.A.K.Indrajith < *@*.***> wrote:
> @anushabhura <https://github.com/anushabhura>
> https://github.com/anushabhura This can be probably your computer ran
> out of storage and it cannot create a checkpoint at 125th step. You can use
> Google co-lab with your google drive. — You are receiving this because you
> were mentioned. Reply to this email directly, view it on GitHub <#149
> <#149>?email_source=notifications&email_token=AKUL53NE6WNCIHECN6PG2D3RBMFHLA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK4TNMI#issuecomment-582563505>,
> or unsubscribe
> https://github.com/notifications/unsubscribe-auth/AKUL53L6OC24H7MUHYXU5JLRBMFHLANCNFSM4DG6R6HQ
> .
>
> Could you show an example command when you executed your code?
> I think, your script need permission to write the checkpoint file.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#149?email_source=notifications&email_token=AKUL53ISUS62J4ZYEWMWKTTRBOIURA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK55G3I#issuecomment-582734701>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AKUL53OHS34AJH4EKRPSKDTRBOIURANCNFSM4DG6R6HQ>
> .
>
|
yes, The checkpoint file should be created in the folder but It needs permission to create a checkpoint file. Please check the permission to create. |
|
May I see your folder in /ckpt/ path. |
|
How can we get permission for creating checkpoint ??
Yeah , my path is attached to screenshots.
…On Thu, Feb 6, 2020 at 11:27 AM Pisit Nakjai ***@***.***> wrote:
May I see your folder in /ckpt/ path.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#149?email_source=notifications&email_token=AKUL53NDA43MGTYKDX7ITHTRBOREZA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK6A67I#issuecomment-582750077>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKUL53L2UVEG2ALFA6UCK23RBOREZANCNFSM4DG6R6HQ>
.
|
|
@anushabhura Sorry, I can't see your attached file. |
|
I am using windows and jupyter( in anaconda navigator).
…On Thu 6 Feb, 2020, 2:53 PM Pisit Nakjai, ***@***.***> wrote:
@anushabhura <https://github.com/anushabhura> Sorry, I can't see your
attached file.
What is your OS system?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#149?email_source=notifications&email_token=AKUL53MOFCCES5C56GSZX6DRBPJHXA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK6PZZY#issuecomment-582810855>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKUL53PYSEDY3YWFX73B7NDRBPJHXANCNFSM4DG6R6HQ>
.
|
|
Check your jupyter is working in your current path. run pwd on the block command in jupyter notebook. |
|
It's " C:\\users\\intel.
…On Fri 7 Feb, 2020, 8:43 AM Pisit Nakjai, ***@***.***> wrote:
Check your jupyter is working in your current path. run pwd on the block
command in jupyter notebook.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#149?email_source=notifications&email_token=AKUL53JKJETJI24ZH3RDZJDRBTGT3A5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELBSKAY#issuecomment-583214339>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKUL53P7A4CG6WL5XMTLU63RBTGT3ANCNFSM4DG6R6HQ>
.
|
|
Do I have to change the working directory so that checkpoint files can be
saved?
…On Fri, Feb 7, 2020 at 10:54 AM Anusha Bhura ***@***.***> wrote:
It's " C:\\users\\intel.
On Fri 7 Feb, 2020, 8:43 AM Pisit Nakjai, ***@***.***>
wrote:
> Check your jupyter is working in your current path. run pwd on the block
> command in jupyter notebook.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#149?email_source=notifications&email_token=AKUL53JKJETJI24ZH3RDZJDRBTGT3A5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELBSKAY#issuecomment-583214339>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AKUL53P7A4CG6WL5XMTLU63RBTGT3ANCNFSM4DG6R6HQ>
> .
>
|
Yes you should |
|
Problem solved . How can I test my trained model ?
On Mon 10 Feb, 2020, 12:29 PM Pisit Nakjai, <notifications@github.com>
wrote:
… Do I have to change the working directory so that checkpoint files can be
saved?
… <#m_-7239818589230966825_>
On Fri, Feb 7, 2020 at 10:54 AM Anusha Bhura *@*.*> wrote: It's "
C:\users\intel. On Fri 7 Feb, 2020, 8:43 AM Pisit Nakjai, @.*> wrote: >
Check your jupyter is working in your current path. run pwd on the block >
command in jupyter notebook. > > — > You are receiving this because you
were mentioned. > Reply to this email directly, view it on GitHub > <#149
<#149>?email_source=notifications&email_token=AKUL53JKJETJI24ZH3RDZJDRBTGT3A5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELBSKAY#issuecomment-583214339>,
> or unsubscribe >
https://github.com/notifications/unsubscribe-auth/AKUL53P7A4CG6WL5XMTLU63RBTGT3ANCNFSM4DG6R6HQ
> . >
Yes you should
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#149?email_source=notifications&email_token=AKUL53K2L4VBR4D3S7KJFG3RCD3OBA5CNFSM4DG6R6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELHN6MI#issuecomment-583982897>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKUL53NFXDFGBIQULOR6VODRCD3OBANCNFSM4DG6R6HQ>
.
|
I have a litle some error with checkpoint.
When I train a model the program save check point in path "./ckpt/cfg/". It work if i load with " --load [numberstep]" but when I want to load last checkpoint with " --load -1 ". The program read checkpoint file in path "./ckpt/". In this path it don't have checkpoint file. The checkpoint file is in ./ckpt/cfg/.
The checkpoint isn't in this folder
when i load with "--load -1"
The text was updated successfully, but these errors were encountered: