Telemetrics daemon is creepy and broken

[nagisa]

When sys.argv[0] is not pointing exactly to the DVC script (which can happen occasionally when wrapping the entry points up) the daemon spawning code will fail as it will try to run $PYTHON $WRAPPER and not just $WRAPPER or $PYTHON $ACTUALPYTHONSCRIPT. $WRAPPER here could be, for instance, a bash script.

---

Not to mention that spawning some daemon to ship data off to the Internets by default is just outright creepy.

[efiop]

Hi @nagisa !

which can happen occasionally when wrapping the entry points up

Could you please describe such a use case in more details?

Not to mention that spawning some daemon to ship data off to the Internets by default is just outright creepy.

Sorry to hear that 🙁 We are spawning it in a detached state to not slow down your dvc commands if your network is not very good. I understand that it might feel a bit invasive, but it really helps us make dvc better. Please see https://dvc.org/doc/user-guide/analytics#anonymized-usage-analytics for a more in-depth explanation.

[nagisa]

Could you please describe such a use case in more details?

A common reason to overwrite argv[0] is to ensure that when user invokes a program, the program appears to act as if it was argv[0]. A good example where this is very relevant is the --help screen. If the wrapper script was:

#/usr/bin/env bash
# some setup here
wrapped_command "$@"

then the the fact that it is a wrapped_command would leak out and the user might mistakenly copy-paste some invocation that misses the wrapper. As the example here shows, overwriting argv[0] might also be necessary to ensure that the program uses the wrapper when it spawns itself.

So instead the script is written as such:

#!/usr/bin/env bash
#some setup here
exec -a "$0" wrapped_command "$@"

and this ensures that this hole gets plugged up, at least to some extent.

In our particular case the wrapper sets up the $PATH and $PYTHONPATH, so missing the wrapper would just make things fail altogether, anyway.

Hope that helps.

[efiop]

@nagisa Do you have some custom wrapper or some well-known solution? We've been using things like brew, pipenv, pyenv, etc, that use different wrappers/shims and never had a problem with those, so I wonder what are you using in your case and why it doesn't work with dvc.

[nagisa]

Well, the wrapper here comes from poetry2nix I believe, but I’m writing this issue here, because I believe its best fixed in dvc.

I can suggest a couple ways to fix this (subjectively ordered from the best to worst):

1. Fork and call the relevant code in the fork;
2. Ignore the error and continue working without a component that’s otherwise optional anyway;
3. Try multiple ways to spawn argv[0] (e.g. try also without prepending sys.executable).

[efiop]

@nagisa So dvc is a dependency in your poetry project, right?

Fork and call the relevant code in the fork;

IIRC, the only way to daemonize on windows is to spawn through the exe, hence why we went that way for compatibility. Though we might consider simply double-forking on linux and mac and keep spawning exe on windows.

Ignore the error and continue working without a component that’s otherwise optional anyway;

Would rather not do that if we can make it work some other way.

Try multiple ways to spawn argv[0] (e.g. try also without prepending sys.executable).

We are doing that for our pyinstaller-built packages, if there is a way to detect that we are currently running in such a way (e.g. maybe there are some env vars that poetry2nix sets that we can base our logic on?) then we could easily do that. Or we could fail and retry, sure.

Sounds like the 3rd option is the simplest to implement. Maybe you would be willing to tweak that piece of code https://github.com/iterative/dvc/blob/0.86.5/dvc/daemon.py#L93 and try it out to see if that solves it for you? 🙂

[nagisa]

We are doing that for our pyinstaller-built packages, if there is a way to detect that we are currently running in such a way (e.g. maybe there are some env vars that poetry2nix sets that we can base our logic on?) then we could easily do that. Or we could fail and retry, sure.

No easy way to check (the wrapper only really adjusts PATH and PYTHONPATH) but dvc could check if the file is valid Python by compileing it first?

[efiop]

@nagisa Compiling seems like it would be too heavy for this application. So maybe simply retry if failed?

[efiop]

Closing due to inactivity.