dekenception: check missing objects from patch and suggest to install libraries

[umlaeute]

so imagine a user downloading a patch from the wild internet. then open it and ... 💥

[list2symbol] couldn't create
[gemwin] couldn't create
[gemhead] couldn't create
[rotate] couldn't create
[curve3d] couldn't create
...

so what now?

it would be great if deken would suggest which libraries are missing and ought to be installed.

@chr15m has started the dekenception branch for this.

[umlaeute]

things to do:

• client-side:
  • find out which objects have failed to create in the patch
• server-side
  • maintain a database of object <-> library mappings
  • query-interface

[umlaeute]

client-side:

last night I created a [findbrokenobjects] object in the iemguts library that searches all top-level patches recursively for un-created objects.
this can be used as a start for a [deken] object.

things to think:

• this functionality can only be implemented as an external
• i haven't found a foolproof way to communicate between the pd-core and pd-gui, that doesn't throw an ugly error if there is no receiver on either side.

the easy part:

• (automatically) set up a receiver as soon as the deken object gets loaded on the core-side.
• (automatically) set up a receiver as soon as the deken-plugin gets loaded on the gui-side.
• communicate between deken and deken-plugin once both have established their receivers.

the hard part:

• when we load the deken-plugin on the gui-side, how do we make sure that deken gets loaded on the core-side? or vice-versa? (but rather not vice-versa, as the gui-side allows us to automatically load a plugin)
• if we cannot trigger a library-loading from the gui-side, how do we check whether the library is already loaded (we also need this if we can trigger the library-loading, in order to ensure that it actually worked).
• how can we do all this without confusing the user with TCL-backtraces (for calling pd plugin-dispatch) and/or deken-plugin: no such object (for calling ::pd_connect::pdsend)

[chr15m]

Good ideas, but maybe in the mean time we can we do like Python's "requirements.txt" instead:

https://pip.readthedocs.org/en/1.1/requirements.html

[danomatika]

I agree that do it within the patch is probably not the right way to go, maintainability and extensibility wise ...

[umlaeute]

how would a requirements.txt differ from an in-patch [declare] statement?

[chr15m]

It took me a while, but I can think of two differences:

1. It's easier to parse (for computers and humans) because all of the dependencies are listed in one place instead of in each separate patch.
2. It can include version numbers.

With regards to (1) it is probably more of a problem in Python where it would be difficult to parse all of the "import" calls without running the scripts. In Pd we are inside the binary and doing things at runtime anyway, and so we can tell whenever a [declare] is loaded, I think, as you outlined in dekenception.

On the other hand, installing the dependencies before loading the patch could be a good idea because you won't get broken object boxes and have to re-load the patches again once the dependencies are installed.

Finally, it is probably useful information for users to see all dependencies listed together in one file.

With regards to (2) I guess that's pretty important looking at long-term use of deken across the life cycle of externals that get updated often. Somebody probably wants to specify which externals are compatible with their patch.

Then again, that gets us into a complicated area because Pd will only load one version. So that would need fairly heavy modifications to allow different versions of the same library to be loaded. Unless we did like Python's "virtualenv" and have an option to install externals in the patch directory instead of e.g. ~/pd-externals.

[umlaeute]

i don't really buy (1), esp. as it requires the author to put dependency information into multiple places, which is errorprone.

(2) might be important though, esp. if it allows us to specify broken versions.

as for supporting installation of multiple versions (not to speak of loading multiple versions): that way lies madness.

[danomatika]

as for supporting installation of multiple versions (not to speak of loading multiple versions): that way lies madness.

Yes.

However, I think there should be some kind of middle ground and, honestly, I think it's the simplest method of all: patch authors should suggest which packages and versions to install. It's up to users of the special packages to install the required externals and deken will already take 90% of the pain out of doing that.

Building an automatic mechanism to handling this for us is admirable, but honestly might not be the best use of energy at this time. It's possible to build a smart system to handle both libs and their versions, but it won't be easy. I think it's far more important to have a working "extended" meta package right now.

[danomatika]

That being said, if there is an easy way to do it, I say go ahead. 💃

I'm just concerned about y'all stretching yourselves too thin in the software development fight of what's possible versus what's actually needed. That also seems to be a problem on the pd-list ...

[chr15m]

@umlaeute I agree that trying to mix multiple library versions in a single environment such as an operating system leads to great difficulty. However I think if it is done in a way where the software in question can only see one of the various versions at any given time (such as in the case of Python's virtualenv, where the Python binary and libraries can only see the one particular version you specify) then it works fine. A lot of the "DLL hell" arose exactly because they don't have access to this type of mechanism. With virtualenv

If Pd had a system whereby you could enter an "externals-free" clean base-state and build up the externals you want in a special environment/mode then this would be equivalent.

In the case of Pd, if the externals are installed in the same folder as the patch then it will prefer that version first, correct?

@danomatika I'm definitely not stretching myself too thin. I've barely worked on deken in weeks. :) Absolutely agree that it's a good idea to pick good bang-for-buck tasks on Free Software projects.

[chr15m]

@umlaeute for deployment this type of mechanism is invaluable. It is often the case that people deploying software (sysadmins, gallery owners) don't know anything about how it works. They just need to get an environment up and running that the software will run in. If they are forced to always have the latest versions of libraries installed then they need to know how to upgrade the software in question to be compatible with the latest versions of the libraries themselves. This can often be an impossible task for somebody without he correct skillset. That is another reason why I think it's good to be able to specify not-the-latest libraries.

[danomatika]

What if you have the option to install specific versions into a local patch folder? Then the author of the patch could bundle the specific libs in cases where this needs to be preserved.

[chr15m]

That is what I am saying, except that the author doesn't do the bundling. They specify which versions of which binaries should be loaded and then the user can use deken to download them optionally to the local patch folder.

[umlaeute]

here's a draft:

dekenception.mp4

1. create an unknown object
2. right-click (or Ctrl-click) the error-message in the Pd-console
3. the deken-dialog will popup
4. the search-field will be pre-filled with the name of the missing object
5. the search mode is set to "objects" only
6. the user must hit Search and then download one of the packages they like.

[umlaeute]

dekenception has made it into Pd-0.53