Add package synchronization mechanism for local/remote python environment

[virtuald]

Idea is something like this:

• User specifies their dependencies in pyproject.toml
• When pyfrc deploys, it ensures your remote environment matches your environment in pyproject.toml
• robotpy-installer looks in your pyproject.toml to know what version to download to your offline cache, and optionally to your local installation too
  • call the command 'sync'?

Vendors are TBD. Probably makes sense to keep vendors in vendordep jsons and integrate that with this somehow... but maybe not.

Python vscode extension would need to know how to update pyproject.toml to upgrade to a specific wpilib wheel. Might just make sense to have that be an installer command?

[virtuald]

Here's another version of the same idea:

well ok, let's go with it for a second -- so let's say you force users to write a pyproject.toml
then you could have some robotpy-specific backend that as part of the build process builds a wheel -- and if there's something in requirements that isn't present in the local cache, it grabs it from the internet
then the deploy thing takes that wheel and sends it to the rio, which automatically installs whatever the requirements are
the deploy thing could probably just build the wheel and deploy it in a single step, it probably wouldn't take very long?
it would just suck if someone updated the requirements and forgot to run a build
but I guess that you'd have that issue anyways?
it's sorta nice too because you could take the wheel and archive it or something to know what you deployed
I think the issue you'll likely run into is someone is going to want to use poetry or whatever the new hotness is
so you'd either have to make it co-exist with that or force users to not do that
I dunno, I think if it were done well it would be pretty neat, and solve some of the issues that we have around requirements
there are probably downsides too

[martinrioux]

Thinking out loud, let me know if I am missing something.
I guess that the smart move is to keep the local installation up to the programmers.
If they use a specialized tool such as poetry, good for them. If they don't, we suggest (and could distribute in some examples) a simple requirements.txt file to make their project more robust to version changes.

There could even be a new "robotpy-installer freeze" command to generate a simple robotpy-requirements.txt file with all robotpy packages.

instead of a "download-pyhon" and "download [packages]", it could be simplified by a "download-all" command witch would do both.
If python is already downloaded, it would simply be skipped. The same goes for all packages.
The local packages version would be used to download the proper rio packages.

Once all is downloaded and you are connected to the rio, I would see a simple "robotpy-install install-all" to install python and all packages (again, based on the local versions). If a robotpy package is found on the local computer but not in the downloaded wheel for the rio, an alert could let the user know that they need to run the "download-all" command again.

Alternatively, all this could even be more automated using within the "robot.py deploy"
The step I would see are:

• Get a list all local robotpy package + python version.
• Check if all of these files are already downloaded, else ask to connect to internet and download them.
• Install everything to in the rio

I guess that for most people, having everything in the deploy command would be the best option.
This could be implemented by creating a "robotpy-install autoinstall" command and having it called from the deploy command.

[virtuald]

Thinking through the workflow a user will take from a clean python installation:

• install minimal dependencies: pip install robotpy

Then they make a robot.py, add the if __name__ == __main__ thing + wpilib.run. Add a pyproject.toml with the following:

[tool.robotpy]

# equivalent to `robotpy==2024.0.0b4`
robotpy_version = "2024.0.0b4"

# equivalent to `robotpy[cscore, ...]`
robotpy_extras = ["cscore"]

# Other pip installable requirement lines
requires = [
  "numpy"
]

Ideally, you'd like them to do:

python robot.py sync [--user]

And that would set up their local environment with the correct pip packages, and also download python and all of the packages for roborio.

However, at least some of the time the user's robot.py would fail because of a missing import, so I think you'd also have to provide a robotpy-installer command that does the exact same thing but doesn't import robot.py:

robotpy-installer sync [--user]

User does normal development, and then they do:

python robot.py deploy

Before doing the deploy...

• It will check to see if python installed on the rio, install it if not
• It will check to see if the local pip packages match whatever is in pyproject.toml and complain if not
  • pip list --format json for inspecting local environment
• It will check to see if the pip packages on the rio match whatever is in pyproject.toml, and if not then it will try to install them (should it delete packages not in pyproject.toml?).
  • Probably should try to detect if the packages are in the local cache?
  • pip list --format json for inspecting remote packages

Then it will do the normal deploy stuff.

The pip sync is going to be expensive to do, so I think it would make sense to cache the synced list on the rio somewhere so that it can be quickly checked (since most of the time it won't need to be done). Probably should store the resulting pip json after install is finished, assuming there's nothing nondeterministic in there.

Probably needs a --no-sync flag.

---

We would also need to support users that don't have a pyproject.toml.

If pyproject.toml didn't exist, I think we would just write one with the current version of robotpy installed in their local environment, then do whatever action that required the pyproject.toml.

---

Python vscode extension would need to know how to update pyproject.toml to upgrade to a specific wpilib wheel. Might just make sense to have that be an installer command?

Probably add robotpy-installer update-robotpy command to do this.

---

The synchronization behavior above is likely covered by pip-tools, but it seems complicated and has a lot of dependencies, so I don't think it's a good candidate for running on the RoboRIO.

[martinrioux]

It all does seem to make sense.
Regarding should it delete packages not in pyproject.toml?, I guess that have a --fresh-install or --reinstall flag to clear old packages would be the simplest.

The only thing I am seeing, is the python robot.py sync [--user]. As you said, should some import be missing, it would prevent things from working. I find it would be easiest to have robotpy-installer sync [--user] part of the workflow when we know we changed a package.

[TheTripleV]

I think it should make a virtualenv and generate a vscode python config. That way packages are project specific like cpp and java

[virtuald]

I think it should make a virtualenv and generate a vscode python config. That way packages are project specific like cpp and java

I would not want to do that by default. I think this could be a separate action setup-vscode-venv or something similar?

[virtuald]

Fixed in #79