Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

users should be warned when they try to start training when a model is already training #136

Open
Abe404 opened this issue Apr 4, 2024 · 4 comments
Assignees

Comments

@Abe404
Copy link
Owner

Abe404 commented Apr 4, 2024

right now if a model is training and the user clicks 'start training' for a different project, training does not start (for the new project) as training is already running, but the user is not told the reason why training does not start.

They should at least be made aware that training was not started because 'training already running for project example_project' or similar.

@Felipegalind0
Copy link
Contributor

I’ll add some code to print a warning on the terminal in the trainer

@Abe404
Copy link
Owner Author

Abe404 commented May 28, 2024

That's a good idea, but in some cases users are running RootPainter purely from the client and only a sysadmin has access to the trainer.

There's a mechanism to send messages to the client that could be used. I'm also wondering if we could add something a little more obvious i.e some type of alert box that pops up in the client telling them that training is already running (the current messages can sometimes be missed, as they get overwritten by the next message).

Just to clarify: The issue is for users that start training for a project where training is already running on the server for a different project. This is an issue that comes up when multiple users are sharing a server.

I'm not 100% sure what the best solution for this is. Basically if users start training (from the client) when the server is already running training for a different user the client should at least let them know what the problem is somehow (in the client, as they might not see the server output).

@Abe404
Copy link
Owner Author

Abe404 commented May 28, 2024

This could also happen in a single user case.

For example, if a user follows the following steps:

  1. User starts training for their project.
  2. User opens a new project
  3. User starts training in the new project (Training won't work because trainer is busy with the previous project but the user doesn't get any feedback in the client).

@Abe404
Copy link
Owner Author

Abe404 commented May 28, 2024

Another option is we rethink how to give the user info about the server status via the client i.e have a custom module (GUI widget) that allows the user to stop/start the server, view server messages, get notifications/warnings/errors from the server etc without looking at the command line. This is a bit more involved and would require more discussion/design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants