-
Notifications
You must be signed in to change notification settings - Fork 449
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LORA] fix model saving and bring into line with other developments in the project #1789
Comments
Now we can set |
We need to implement the S3 artifact store to support saving large files remotely later and support saving folders directly to artifact store. |
Idea to add "saving a directory" to |
@jieguangzhou to add proposal for saving list of artifacts in model. |
Original Artifact workflowInput:
Loading
New Artifact workflow with saving a directoryNew DataType instance: file x is path or directory encode(x): check x exist and return x decode(x): check x exiet and return x SavingInput:
Loading
Otherrename the key |
Can we go one level higher. Am I correct in thinking you want to build a local cache/ copy of some content in the artifact store? My question is, is this necessary? Especially, for instance, if we are using What problem are we solving here? |
If we use FileSystemArtifactStore, we can use symlink or copy a new directory. But if we only use symlink, the directory will not actually be saved in artifact_store. For example: If I want to training a model on server1, and deploy the service on server2. We need copy the whole directory of I think all the artifacts need to save into
This solves the problem that |
@jieguangzhou I agree with the general proposal. What will the schema inside Also, how will we synchronize directories to the artifact store? With, for instance, |
the
I posted a quick implementation for this, please help to take a look. #1805 @blythed |
Currently model saving is not happening in the same or a similar way to other models. We should use the artifact store.
If this doesn't work for some reason we should evaluate a better method.
The text was updated successfully, but these errors were encountered: