New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package Manager #123

Open
mottosso opened this Issue Oct 14, 2014 · 15 comments

Comments

Projects
None yet
3 participants
@mottosso
Member

mottosso commented Oct 14, 2014

Goal

To simplify and encourage building, sharing and installing additions to Pyblish; most notably validators, extensions and integrations.

In the future, I'd like for there to be a repository of validators, written by anyone, to be reused at your facility. Like if someone had already written that "inverted normals" validator, just reuse it. The same goes for selectors, extractors and conformers; each plug-in can then be properly documented on it's own GitHub/BitBucket page and mixed in with other plug-ins for a combination unique to your requirements.

Architecture

A permanent web-based app will host available components of Pyblish and allow each to be inspected and installed locally through the same set of commands. Similar to:

$ pyblish install validate-inverted-normals
$ pyblish install extract-maya-ascii

At the moment, one realistic way of browsing available components of Pyblish is by simply looking at PyPI for anything that starts with "pyblish"

$ pip search pyblish
pyblish                   - Plug-in driven automation framework for content
pyblish-nuke              - Nuke integration of Pyblish
pyblish-napoleon          - Asset Framework for Pyblish
pyblish-maya              - Maya integration of Pyblish

An alternative method may be for each plug-in author to manually register with the package management web app.

@mottosso mottosso added this to the 1.2 milestone Oct 14, 2014

@tokejepsen

This comment has been minimized.

Show comment
Hide comment
@tokejepsen

tokejepsen Apr 23, 2015

Member

Maybe searching github for repos starting with pyblish and search the plugins folder within. Maybe you can tag files with git for search words.

Member

tokejepsen commented Apr 23, 2015

Maybe searching github for repos starting with pyblish and search the plugins folder within. Maybe you can tag files with git for search words.

@tokejepsen

This comment has been minimized.

Show comment
Hide comment
@tokejepsen

tokejepsen Apr 23, 2015

Member

Also guess we need a system for having a default plugins folder for Pyblish that the "plugins install" system can store them in.

Member

tokejepsen commented Apr 23, 2015

Also guess we need a system for having a default plugins folder for Pyblish that the "plugins install" system can store them in.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Apr 23, 2015

Member

Maybe searching github for repos starting with pyblish and search the plugins folder within.

Believe it or not, but this was just how the first prototype worked. Searching GitHub is very fast and well suited for the task.

Maybe you can tag files with git for search words.

I like the generality of that suggestion, but it might be a tad too general. Ideally each distribution would carry some amount of metadata, like author, version and a README, such that we could build a summary similar to the Sublime Text Package Control.

Also guess we need a system for having a default plugins folder for Pyblish that the "plugins install" system can store them in.

That's true. Spontaneously, I'm thinking we could add a default search location for plug-ins in the users home directory, that can be overridden somehow, possibly via an environment variable.

Each download would by default end up local to the user installing it, and for more advanced use, it'd end up wherever the override said it should. This is how Sublime Text works and I think it's quite intuitive, though I haven't tried it in any distributed fashion as would probably most often be the case for a studio install of Pyblish.

Member

mottosso commented Apr 23, 2015

Maybe searching github for repos starting with pyblish and search the plugins folder within.

Believe it or not, but this was just how the first prototype worked. Searching GitHub is very fast and well suited for the task.

Maybe you can tag files with git for search words.

I like the generality of that suggestion, but it might be a tad too general. Ideally each distribution would carry some amount of metadata, like author, version and a README, such that we could build a summary similar to the Sublime Text Package Control.

Also guess we need a system for having a default plugins folder for Pyblish that the "plugins install" system can store them in.

That's true. Spontaneously, I'm thinking we could add a default search location for plug-ins in the users home directory, that can be overridden somehow, possibly via an environment variable.

Each download would by default end up local to the user installing it, and for more advanced use, it'd end up wherever the override said it should. This is how Sublime Text works and I think it's quite intuitive, though I haven't tried it in any distributed fashion as would probably most often be the case for a studio install of Pyblish.

@tokejepsen

This comment has been minimized.

Show comment
Hide comment
@tokejepsen

tokejepsen Apr 23, 2015

Member

I like the generality of that suggestion, but it might be a tad too general. Ideally each distribution would carry some amount of metadata, like author, version and a README, such that we could build a summary similar to the Sublime Text Package Control.

Just to be clear, I'm talking about installing individual plugins instead of the packages of plugins. Definitely should be the choice of installing packages as well, but taking only the one plugin from a package gives us the ability to not have to change a complete pipeline for the sake of one plugin to work.

Member

tokejepsen commented Apr 23, 2015

I like the generality of that suggestion, but it might be a tad too general. Ideally each distribution would carry some amount of metadata, like author, version and a README, such that we could build a summary similar to the Sublime Text Package Control.

Just to be clear, I'm talking about installing individual plugins instead of the packages of plugins. Definitely should be the choice of installing packages as well, but taking only the one plugin from a package gives us the ability to not have to change a complete pipeline for the sake of one plugin to work.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Apr 23, 2015

Member

Yes, I am also talking about a single plug-in.

GitHub repositories are a great candidate for this, they are lightweight and hosts a user-friendly landing page for any plug-in, and also provides revision control, tracker and wiki. But in the simplest of cases, the repository simply contains a single file, which is the plug-in itself.

Have a look at how simple some of the Sublime Text plug-ins can get.

https://github.com/demisx/sublime-vim-navigation

Member

mottosso commented Apr 23, 2015

Yes, I am also talking about a single plug-in.

GitHub repositories are a great candidate for this, they are lightweight and hosts a user-friendly landing page for any plug-in, and also provides revision control, tracker and wiki. But in the simplest of cases, the repository simply contains a single file, which is the plug-in itself.

Have a look at how simple some of the Sublime Text plug-ins can get.

https://github.com/demisx/sublime-vim-navigation

@tokejepsen

This comment has been minimized.

Show comment
Hide comment
@tokejepsen

tokejepsen Apr 23, 2015

Member

alright, thats cool. We could have a package.json that points to where the plugins are relative to the repo. Possibly with tag data as well.

Member

tokejepsen commented Apr 23, 2015

alright, thats cool. We could have a package.json that points to where the plugins are relative to the repo. Possibly with tag data as well.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Apr 23, 2015

Member

Sounds doable

Member

mottosso commented Apr 23, 2015

Sounds doable

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Apr 23, 2015

Member

Believe it or not, but this was just how the first prototype worked. Searching GitHub is very fast and well suited for the task.

Actually I remembered it wrongly, it was searching PyPI using pip search. Searching GitHub should be much better, and doesn't require uploading to PyPI.

Member

mottosso commented Apr 23, 2015

Believe it or not, but this was just how the first prototype worked. Searching GitHub is very fast and well suited for the task.

Actually I remembered it wrongly, it was searching PyPI using pip search. Searching GitHub should be much better, and doesn't require uploading to PyPI.

@tokejepsen

This comment has been minimized.

Show comment
Hide comment
@tokejepsen

tokejepsen Apr 23, 2015

Member

Heres a package json file from an atom package; https://github.com/rgbkrk/atom-script/blob/master/package.json

Member

tokejepsen commented Apr 23, 2015

Heres a package json file from an atom package; https://github.com/rgbkrk/atom-script/blob/master/package.json

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Apr 23, 2015

Member

That looks very good. As it includes the link to the repository, it effectively decouples the json from the repository, meaning we could simply have a collection of those json files some place, and host the actual plug-in and data wherever, including GitHub.

It also includes dependencies, which opens some interesting doors, if for example a single plug-in depends on Pyblish QML, or even a larger library such as Magenta or Napoleon.

Member

mottosso commented Apr 23, 2015

That looks very good. As it includes the link to the repository, it effectively decouples the json from the repository, meaning we could simply have a collection of those json files some place, and host the actual plug-in and data wherever, including GitHub.

It also includes dependencies, which opens some interesting doors, if for example a single plug-in depends on Pyblish QML, or even a larger library such as Magenta or Napoleon.

@dshlai

This comment has been minimized.

Show comment
Hide comment
@dshlai

dshlai Jan 4, 2016

Here is how Julia does it (for your consideration):
https://github.com/JuliaLang/METADATA.jl

/Package/ # Package Name
/versions/ # root folder of the versions
/0.x.x # each version of the package
url # url text file that points the git landing page of that package

dshlai commented Jan 4, 2016

Here is how Julia does it (for your consideration):
https://github.com/JuliaLang/METADATA.jl

/Package/ # Package Name
/versions/ # root folder of the versions
/0.x.x # each version of the package
url # url text file that points the git landing page of that package

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Jan 4, 2016

Member

Thanks @dshlai, the more reference the better!

And while we're at it, I implemented a Sublime Text equivalent on this for be and found it worked very well.

In a nutshell, users create a be preset locally, append a json configuration file to it, and insert the link to their GitHub repository to be-presets and submit a pull-request for it to be automatically included on the site and downloadable via the CLI.

Member

mottosso commented Jan 4, 2016

Thanks @dshlai, the more reference the better!

And while we're at it, I implemented a Sublime Text equivalent on this for be and found it worked very well.

In a nutshell, users create a be preset locally, append a json configuration file to it, and insert the link to their GitHub repository to be-presets and submit a pull-request for it to be automatically included on the site and downloadable via the CLI.

@dshlai

This comment has been minimized.

Show comment
Hide comment
@dshlai

dshlai Jan 4, 2016

Which you guys think is better? Centralized metadata repo (aka presets.json, or like Julia's METADATA.jl) or searchable prefix (aka pyblish_xxxx)?

Seems like centralized repo is easier to maintain, but does require a maintainer

dshlai commented Jan 4, 2016

Which you guys think is better? Centralized metadata repo (aka presets.json, or like Julia's METADATA.jl) or searchable prefix (aka pyblish_xxxx)?

Seems like centralized repo is easier to maintain, but does require a maintainer

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Jan 4, 2016

Member

Seems like centralized repo is easier to maintain, but does require a maintainer

The Package Control of Sublime Text is run by a single maintainer, and though it looks to work fine, I share your sentiment in that it does mean that if he dies or otherwise stops showing interest in what he's doing, we're all doomed.

However, even with his approach, and the approach I adopted for be, the centralized repo is a GitHub repo, and it can have multiple maintainers, and most importantly (IMO) pull-requests for inspection of every package.

One of the dangers of letting anyone submit and expose anything, is that you can't be too sure whether what you download is functional, appropriate, duplicate or even something like a virus. A centralised repo, and having oversight over what is included I think is essential to any distributed collection of software.

Member

mottosso commented Jan 4, 2016

Seems like centralized repo is easier to maintain, but does require a maintainer

The Package Control of Sublime Text is run by a single maintainer, and though it looks to work fine, I share your sentiment in that it does mean that if he dies or otherwise stops showing interest in what he's doing, we're all doomed.

However, even with his approach, and the approach I adopted for be, the centralized repo is a GitHub repo, and it can have multiple maintainers, and most importantly (IMO) pull-requests for inspection of every package.

One of the dangers of letting anyone submit and expose anything, is that you can't be too sure whether what you download is functional, appropriate, duplicate or even something like a virus. A centralised repo, and having oversight over what is included I think is essential to any distributed collection of software.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 13, 2016

Member

With the addition of support for instances with multiple families, we should have another go at this!

Member

mottosso commented Feb 13, 2016

With the addition of support for instances with multiple families, we should have another go at this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment