Flamenco error after restart: "Error fetching hashed password error="not found""

[jhowe-3d]

The computer that I was running the manager on was restarted and Flamenco was not shut down before that happened. I cannot get Flamenco running again where it was working well before. The manager, when run in verbose mode shows the following error

�[36mINFO�[0m[2018-02-26T02:12:27-05:00] SendStartupNotification: trying to send notification.
�[33mWARN�[0m[2018-02-26T02:12:27-05:00] Error fetching hashed password                �[33merror�[0m="not found"
�[36mINFO�[0m[2018-02-26T02:12:28-05:00] Timer 'StartupNotifier' goroutine shutting down.
�[36mINFO�[0m[2018-02-26T02:12:28-05:00] SendStartupNotification: Done sending notification to upstream Flamenco
�[33mWARN�[0m[2018-02-26T02:12:28-05:00] Error fetching hashed password                �[33merror�[0m="not found"
�[33mWARN�[0m[2018-02-26T02:12:28-05:00] Error fetching hashed password                �[33merror�[0m="not found"
�[33mWARN�[0m[2018-02-26T02:12:28-05:00] Error fetching hashed password                �[33merror�[0m="not found"

I ran the local manager setup again after unzipping a new copy of the manager install into a new folder. It connects to cloud.blender.org and lets me select a manager there. I revoked the token and recreated the config using the defaults.

Flamenco workers on the local network appear to be trying to send information if they are started up, with the following errors appearing in the worker command prompt.

2018-02-26 02:17:39,402     INFO flamenco_worker.upstream_update_queue.TaskUpdateQueue Pushing task update to Manager
2018-02-26 02:17:39,402  WARNING flamenco_worker.worker.FlamencoWorker Unable to POST to manager /sign-on, retrying in 30 seconds: 401 Client Error: Unauthorized for url: http://192.168.1.207:8083/sign-on
2018-02-26 02:17:39,402  WARNING flamenco_worker.upstream_update_queue.TaskUpdateQueue Manager did not accept our updates (401 Client Error: Unauthorized for url: http://192.168.1.207:8083/tasks/5a937b5c943c61270d70e4fa/update), will retry later.

[sybrenstuvel]

It looks like you didn't copy Flamenco Manager's database to the new copy. As a result, it lost the worker credentials. In other words, the Worker is trying to log in with an account that doesn't exist any more. Run the worker with the --reregister CLI argument to make it forget its credentials and register a new account.

[sybrenstuvel]

I've made a ticket for this issue, as it's quite common that people get this error. We can make it easier in a future release ;-)

[jhowe-3d]

Sounds good. Reregistering the Workers will likely eliminate some errors. In my case, I had to also go in and run the manager with --cleanslate option to get everything going again.

[sybrenstuvel]

In my case, I had to also go in and run the manager with --cleanslate option to get everything going again.

Which error did that fix for you?

[jhowe-3d]

I had many errors, including some missing task errors and worker task updates that were not being pushed up to the server, but this is because I had made a mess after the reboot. Nothing was working, so I deleted projects on the server (but still had some jobs in them), created a new and separate manager, purged out stored jobs, and deleted worker databases in an effort to get things working again.

[sybrenstuvel]

Ah, that explains the error Manager 5a93b20d943c612870f31e28 sent update for task 5a93a6ca943c612811d957aa which belongs to other manager 5a8999ae23b3642868b6b50c we see in our logs.

I prefer to do things cleanly and just clean up a mess, rather than deleting stuff and starting over, as it makes things a lot easier in the long run. Feel free to contact us here or at cloudsupport@blender.org for any support you need.

[jhowe-3d]

Doing things cleanly is ideal, but don't underestimate the sheer panic when your render farm goes down and won't come back up. I have done a LOT of learning about Flamenco in the past week!

[sybrenstuvel]

Yeah, I can appreciate the terror ;-)