The future of the project commands

[smsegal]

I'm currently using the project functionality to synchronize multiple projects to a remote server. This makes it much easier than starting separate sync sessions for each of the projects, and lets me specify the alpha/beta endpoints in the yaml file.

Since docker-compose isn't an option for our infra, would you recommend any other built-in functionality that can support the same use-case now that project is deprecated? If not, I suppose a simple shell-script wrapping a bunch of calls to mutagen sync is the way to go.

ref: #356

[xenoscopic]

@smsegal Thanks for reaching out. I can hold off on removal for the time being. The code there isn't too much of a maintenance burden, but given that it's fallen behind a bit in terms of features, I'd like to get a better plan-of-action for its continued maintenance. I don't really use it personally enough to guide its evolution. It probably does make sense to continue maintaining something like it, but it might be better off as an external project.

Just to spark some discussion, would your use case be able to adjust to:

• An external mutagen-project command? (Name TBD... that's sort of a mouthful)
• A slightly different (more Compose-ish) lifecycle model (i.e. up/down commands) that's more extensible?
• A different file name (to make mutagen.yml a bit less overloaded?)

If you can provide a bit more info about how you're using projects right now, it would be a big help. I wasn't actually sure if anybody was still using them (since I keep Mutagen analytics/telemetry-free).

Doing some shell scripts could be an option; that was certainly the recommended route pre-Mutagen Compose, but I'm still amenable to a more structured solution.

[smsegal]

Sure, I can lay out our specific use-case in some more detail.

We use nomad to schedule interactive and batch sessions on our GPU cluster (each session is a container) to do ML training and experimentation.

We have several libraries that we use and develop simultaneously as part of our training process, and so we're using mutagen project to sync our working copies of the libraries and the training repo from our local dev machines to (a container on) the remote cluster.

What this really boils down to in terms of mutagen functionality is a mutagen.yml file in a developers projects folder (for example) that contains entries for each of the projects that need to be synced to the remote session.

sync:
  defaults:
    ...
  # these two blocks for each library and project
  project-code:
    mode: "one-way-safe"
    alpha: "./ml-project"
    beta: "remote-url:/storage-location/ml-project"
    ignore:
      vcs: true
  project-vcs:
    mode: "one-way-replica"
    alpha: "./ml-project/.git"
    beta: "remote-url:/storage-location/ml-project/.git"
    ignore:
      paths:
        - "index"

• An external mutagen-project command? (Name TBD... that's sort of a mouthful)
• A slightly different (more Compose-ish) lifecycle model (i.e. up/down commands) that's more extensible?
• A different file name (to make mutagen.yml a bit less overloaded?)

I think all of those are fairly easy to adapt to. For the first suggestion, I'd consider implementing a behaviour similar to git, where if a command starting with git- (e.g git-mycommand) is in the $PATH, then it can be called by git mycommand. That should make it very easy to drop in external commands.

The compose-ish model also works, although our specific use case doesn't (currently) take advantage of any potential functionality that might provide.

And finally, a different file name is also very easy to deal with as well :)

[xenoscopic]

Cool, thanks for the additional details.

Out of curiosity, are you using any of the (before|after)(Create|Terminate) hooks in projects?

I'll leave projects in place for now, though I'm probably going to leave them marked as deprecated, just to set expectations. However, since people are still using them, I'll put the removal on hold until there's a better migration story (and hopefully a trivial one).

I don't really have the time (at least not at the moment) to extract projects into a separate tool, but I'll revisit that idea in a few months once the Mutagen daemon API has been refactored and external maintenance is more feasible. I suppose projects can serve as a test bed for that public API if nothing else.

[smsegal]

No, we're not using any of the hooks - It's something I looked at, but our infra doesn't support it since we don't actually connect over SSH, and don't open those ports generally.

[asan127]

+1 on not removing project. We just need a means to start multiple syncs and forwardings, and we're not in a place to try and make the compose work any time soon.

[xenoscopic]

I've changed the title of the issue for now because I think this is better off as a discussion about how to move forward.

At this point in time, I don't plan to remove the project functionality without a suitable replacement, and I have updated the release notes for v0.15 to reflect that. Sorry for the scare.

Barring a desire to evolve the way projects work, I think the plan of action is therefore to leave projects in place until the Mutagen daemon API stabilizes, then extract them into a separate project (similar to Mutagen Compose) and just let people continue using them that way.

Of course, I'd love to see some feature improvement there, so I'll leave this issue open in case people want to chime in with ideas.

[yourfrienddhruv]

+1 on keeping this functionality. Either In core or as an extension. Fine with having any file name/format or multiple files.

To give details on our workflow:

Each team member is responsible for a specific service in a large server-client-peer software where lots of ports are used for IOT data exchange. So as developer of a specific services mutagen provides easy way to setup other services on remote(port mapped to local) and while debugging own service on local.

Every team member has own "remote-server" and mutagen.yml allows easy collaboration on all common tasks to be done on remote and sometime local. All members commit all frequently used commands in this central file and everyone having own ssh-alias pointing to own server directly uses these project commands

As I definitely don't want to be mean and ask the Author to keep this feature in core, but would really wish this can be done in one or other way.

Thank you again to Author and community for the great gem of engineering. We spent a year+ on various personal-dev-sanboxes efforts including web-ide efforts (paid + open) and finally this was life saver. and showing mutagen.yml to a new member was enough to get a new guy on-boarded.

[pgsimon]

+1 for keeping mutagen project functionality. Long story as short, we are using this for remote syncing.

[dev-zero]

+1 on keeping the mutagen project functionality. While container support is being added in the HPC world, most things still happen on the login nodes. Being able to simply copy in a configuration file to a local source tree and get things synced to wherever needed is very useful and much in-line with the rest of our workflow.

[pepawel]

+1 for keeping mutagen project feature. Either in a core tool or as something external.

[xenoscopic]

Thanks for all of the use case information. It sounds like there's enough demand to keep this feature around and to invest some more time into it. I don't have an exact timeline or plan for revisiting development on projects, but I'll keep them working as-is for now and remove the "deprecated" mark in an upcoming release (probably v0.15.1 this weekend). I'll leave this issue open as a discussion point in case people have any ideas about how they might want to see this functionality evolve (if at all).

I still think migrating this functionality to an external project would (a) be a great test case of the daemon API and (b) allow this functionality to evolve more easily. In that case, we can also just alias mutagen project to that external command, so it should be doable without any break to scripts or external workflows.

[xenoscopic]

The mutagen project commands are now officially un-deprecated with Mutagen v0.15.1. Thank you all for the feedback.

[j0eii]

the mutagen project functions are very useful when i use it to manage multiple packages.

I also suggest mutagen should also implement alias for
mutagen project terminate = mutagen pj t etc
that will be very useful under windows environment.

[beatcracker]

My 2¢.

I just recently started using mutagen to sync the source code from Windows to WSL2 via SSH. This enables me to use most of the tools I like on the Windows side and then do some heavy IO processing in the WSL.

I started with the plain mutagen sync create but then switched to the project since it's much easier to have a per-folder setup.

Here's what I use:

# Make sure that SSH server is up in WSL
beforeCreate:
  - wsl.exe --distribution Ubuntu --user root --exec /usr/sbin/service ssh start

sync:
  defaults:
    flushOnCreate: true
  code:
    alpha: 'c:/Company/Repo'
    beta: 'user@localhost:2222:~/mutagen/company-repo'
    mode: 'two-way-resolved'
    ignore:
      paths:
        # Terraform doesn't work well with x-platform caches
        - '.terraform/'

[indigoviolet]