Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to add Instances and have their families' Plugins show after collection #359

Open
BigRoy opened this issue Jan 31, 2020 · 13 comments · May be fixed by pyblish/pyblish-qml#356
Open

Allow to add Instances and have their families' Plugins show after collection #359

BigRoy opened this issue Jan 31, 2020 · 13 comments · May be fixed by pyblish/pyblish-qml#356

Comments

@BigRoy
Copy link
Member

BigRoy commented Jan 31, 2020

Issue

I'm in the process of setting up a Publishing workflow where some instances will spawn "dependent publishes" if they are set to publish in the current run, e.g. the user did not disable them in the Pyblish UI. More info on what I'm trying to achieve can be found in this Avalon + USD + Pyblish discussion.


We've recently started looking into adopting Houdini 18 Solaris with USD inside Avalon and are making good progress. Currently we have a good view of how we want to structure assets and shots with a lot of information gained from this Generating Assets and Shots in USD (Pipeline) topic.

Use case

  • We have a couple of model variations the User may want to publish. These are now showing as our Publish instances.
  • Whenever any Model variation instance was active when the user starts the Publish two dependent "publishes" should be created to, specifically a master usdModel and usdShade.
  • Whenever any usdModel file is to be generated it should also bootstrap an usdAsset file whenever the usdAsset file does not exist yet.

The crucial point is that these dependencies are versioned seperately and turn into their own subset.

So e.g.

  • Artist generated model variation usdModel_damaged
  • This means master usdModel and usdShade will need to be updated in their respective subset and versions.
  • If usdAsset does not exist yet (e.g. this is first usdModel ever) then also generate usdAsset subset v001.

I've found it particularly hard to make these dependencies simple to write and debug inside Pyblish and Avalon. This commit has the crucial bits of my current "quick 'n' dirty" draft.


As such, after the instances are collected and the user made his pick as to what to publish then when the user clicks validate or publish the dependent instances get created. However, currently these will not themselves have their relevant plug-ins loaded for their family. This is currently due to how after Collecting the irrelevant Plug-ins are removed. For example this second Validator will never trigger:

class CreateDependencies(pyblish.api.ContextPlugin):
    order = pyblish.api.ValidatorOrder - 0.4

    def process(self, context):
        instance = context.create_instance("dependency")
        instance.data["family"] = "dependency"


class ValidateDependencies(pyblish.api.InstancePlugin):
    order = pyblish.api.ValidatorOrder
    families = ["dependency"]

    def process(self, instance):
        # Never runs...
        self.log.info("Running dependency")

This means I cannot add new instances or alter the instances after the User's publishing choices are well defined. Because I can't add it in-between Collecting and Validating.

Changing CVEI?

In the discussion I opted for maybe adding an additional step to CVEI, specifically:

  1. Initialize - find and create the Instances for the user to interact with, only with essential information to allow the user to check the contents.
  2. Collect - collect any additional data needed for validation, extraction and integration. Including the potential adding of automated dependent instances.
  3. Validate - validate all the existing instances
  4. Extract - output the publish instances.
  5. Integrate - push the extracted data into the correct location on disk and into your production database.

A user would then be able to "change what instances to trigger" (e.g. through Pyblish QML) at the end of Initialize (1). Then on Validate or Publish the publishing will continue onwards through respectively 2-3 or 2-5. The difference in behavior would then be after Collect (2) that all plugins are still checked whether they should be run for the potentially new or changes families.


I would love to see a way where I can trivially add instances as dependencies after the user initiated the publish choices and still allow them to be shown individually to the user as a result. This would then also allow to debug those dependent publishes right in the Pyblish User Interfaces.


This is somewhat related to #346 due to the allowing the Initialize step to become solely the "list which instances are available to publish" and all the post collecting of data to be done in then step two: Collect.

@mottosso
Copy link
Member

mottosso commented Feb 1, 2020

More info on what I'm trying to achieve can be found in this Avalon + USD + Pyblish discussion.

In the interest of avoiding broken links in the future, could you ensure the contents are included here, maybe as a <detail></detail> snippet?

@BigRoy
Copy link
Member Author

BigRoy commented Feb 1, 2020

In the interest of avoiding broken links in the future, could you ensure the contents are included here, maybe as a snippet?

Updated the original comment to include the crucial details.


Regarding solutions. I tried another one but it didn't work either.

What I tried was:

import pyblish.api

def on_instance_toggled(instance, new_value, old_value):
    """Pyblish Instance toggled dependencies callback.

    This will check if Instance has a function `on_toggled()` and when present
    will call it respectively. This allows Instances to produce custom toggle
    behavior.

    """

    if not hasattr(instance, "on_toggled"):
        return

    instance.on_toggled(new_value, old_value)

pyblish.api.register_callback("instanceToggled", on_instance_toggled)

Which would allow me to add custom toggle callbacks per Instance. So then I added my toggle callback:

# psuedocode - this should be in ContextPlugin.process()
import types

def _on_toggle(self, new_value, old_value):
    for instance in self.data["dependencies"]:
        # Match the state on dependent instances
        instance.data["publish"] = new_value

instance = context.create_instance("foobar")
# Add our on_toggled method
instance.on_toggled = types.MethodType(_toggle, instance)

Now in Pyblish QML when toggling the Instance the callback triggers as expected. However in the end it still presents two issues:

  • The updates to the dependent instances' states is not reflected in the UI. I guess it does not explicitly get that change refreshed towards the UI.
  • The Collector order had already run, so even I'd enable new instances those new Plug-ins would never appear I guess. Similarly with disabling would that ever make the related Plug-ins get removed visually? I assume it is possible to get to that behavior since toggling all instances off of a family makes their related plug-ins disappear too.

This makes me think having an Order available for post-processing the Plugins after the first step (e.g. Initialize -> Collect as described in the first comment).

Other ideas?

@tokejepsen
Copy link
Member

Now in Pyblish QML when toggling the Instance the callback triggers as expected. However in the end it still presents two issues

Seems like you might be onto something here, instead of introducing a new step to CVEI. Does this work with pyblish-base alone then? If so its "just" a UI question.

@BigRoy
Copy link
Member Author

BigRoy commented Feb 2, 2020

Does this work with pyblish-base alone then? If so its "just" a UI question.

Well, pyblish-base has no knowledge of a instance toggled callback. Right? The toggling happens based on a UI callback. However, even if after Collect I could have it to change the dependent instances (e.g. activate/deactivate them for publishing) then it wouldn't work because the code filters out plugins directly after Collect, right?

@tokejepsen
Copy link
Member

Well, pyblish-base has no knowledge of a instance toggled callback. Right?

Sorry, yeah. Meant whether you could enable an instance after collection and have the correct plugins run.

the code filters out plugins directly after Collect, right?

I thought that code ran after the collection stage? So if any instance's active stage has changed (with for example your instanceToggled callback) then the related plugins will run.
(Also note that this is not the iterator pyblish-qml runs, its this)

from pyblish import api, util


class CreateDependencies(api.ContextPlugin):
    order = api.CollectorOrder

    def process(self, context):
        instance = context.create_instance("dependency")
        instance.data["family"] = "dependency"
        instance.data["publish"] = False


class ValidateDependencies(api.InstancePlugin):
    order = api.ValidatorOrder
    families = ["dependency"]

    def process(self, instance):
        print("Running dependency")


api.register_plugin(CreateDependencies)
api.register_plugin(ValidateDependencies)

# Will not run ValidateDependencies.
util.publish()

# Will run ValidateDependencies.
context = util.collect()

# User interaction happens here.
context[0].data["publish"] = True

util.validate(context)

@BigRoy
Copy link
Member Author

BigRoy commented Feb 2, 2020

Your example seems to be the wrong use case compared to what I'm describing. So yes, if the user enables or disables existing instances of specific families that works - that's what we've all been doing so far. But in this case I have an Instance that has Instances that are not to be toggled by the user but they are instances that toggle depend on whether this (and/or some others) are active.

from pyblish import api, util


class CreateVariationsWithDependency(api.ContextPlugin):
    order = api.CollectorOrder

    def process(self, context):

        variations = ["default", "open", "big"]
        for variation in variations:
            instance = context.create_instance(variation)
            instance.data["family"] = "usd"
            instance.data["publish"] = False
            
        # When any of the variation instances are active
        if any(varations):
            instance = context.create_instance("dependency")
            instance.data["family"] = "dependency"
            instance.data["publish"] = False
            
            # Do not allow user to toggle, but have its active state 
            # depend on whether at least one of the variations is
            # about to be published.
            instance.data["optional"] = False  
            


class ValidateDependencies(api.InstancePlugin):
    order = api.ValidatorOrder
    families = ["dependency"]

    def process(self, instance):
        print("Running dependency")


api.register_plugin(CreateVariationsWithDependency)
api.register_plugin(ValidateDependencies)

So it would be something like this. In that scenario... or with a use case like that - how do I make that happen? The dependency would run once no matter how many variations are active, as long as there's at least one. (In my actual use case it would run once per all model variations for one asset, so it would be one level further down.)

@tokejepsen
Copy link
Member

Sorry if I'm misunderstanding your case. This is how I see the issue solved:

from pyblish import api, util


class CreateVariations(api.ContextPlugin):
    order = api.CollectorOrder

    def process(self, context):

        variations = ["default", "open", "big"]
        for variation in variations:
            instance = context.create_instance(variation)
            instance.data["family"] = "usd"
            instance.data["publish"] = False


class CreateDependencies(api.ContextPlugin):
    order = api.CollectorOrder

    def process(self, context):
        instance = context.create_instance("dependency")
        instance.data["family"] = "dependency"
        instance.data["publish"] = False

        # Do not allow user to toggle, but have its active state
        # depend on whether at least one of the variations is
        # about to be published.
        instance.data["optional"] = False


class ValidateDependencies(api.InstancePlugin):
    order = api.ValidatorOrder
    families = ["dependency"]

    def process(self, instance):
        print("Running dependency...")


class ValidateUSD(api.InstancePlugin):
    order = api.ValidatorOrder
    families = ["usd"]

    def process(self, instance):
        print("Running \"{}\" instance...".format(instance))


api.register_plugin(CreateVariations)
api.register_plugin(CreateDependencies)
api.register_plugin(ValidateDependencies)
api.register_plugin(ValidateUSD)

context = util.collect()

# User enables "default" instance.
[x for x in context if str(x) == "default"][0].data["publish"] = True
# 'instanceToggled' callback runs.
instances = [x for x in context if x.data["family"] == "dependency"]
for instance in instances:
    instance.data["publish"] = True

util.validate(context)

@davidlatwe
Copy link
Contributor

Just to make sure that I followed the issue right, does the following example able to describe your end goal (use case) ?

import pyblish.api
import pyblish_qml


# ==============================================
# Collecting (or Initializing) phase
class CreateToy(pyblish.api.ContextPlugin):
    order = pyblish.api.CollectorOrder - 0.1

    def process(self, context):
        instance = context.create_instance("Bear")
        instance.data["family"] = "toy"
# Pyblish GUI stop at this point and wait for user input,
# like toggling on/off "toy" instances.

# ==============================================
# Post-Collect (or Collecting) phase, when user pressed publish/validate button
# Create more instance or changing context based on user input.
class CreateToyBox(pyblish.api.ContextPlugin):
    order = pyblish.api.CollectorOrder + 0.4

    def process(self, context):
        for instance in list(context):
            if not instance.data["family"] == "toy":
                continue
            name = instance.name + "Box"
            parent = context.create_instance(name)
            parent.data["family"] = "toyBox"


class CollectToyBoxData(pyblish.api.InstancePlugin):
    order = pyblish.api.CollectorOrder + 0.41
    families = ["toyBox"]

    def process(self, instance):
        self.log.info("Collecting ToyBox data from %s." % instance)


# ==============================================
# Validating and the rest
class ValidateToy(pyblish.api.InstancePlugin):
    order = pyblish.api.ValidatorOrder
    families = ["toy"]

    def process(self, instance):
        self.log.info("Validating Toy.")


class ValidateToyBox(pyblish.api.InstancePlugin):
    order = pyblish.api.ValidatorOrder
    families = ["toyBox"]

    def process(self, instance):
        self.log.info("Validating Toy Box.")

@BigRoy
Copy link
Member Author

BigRoy commented Feb 5, 2020

@davidlatwe that does sound like the solution I described and seems to solve my use case, yes. As long as the crucial comment is taken into account as you wrote it:

# Pyblish GUI stop at this point and wait for user input,
# like toggling on/off "toy" instances.

@davidlatwe
Copy link
Contributor

I think I have got a working draft :

post-collect

@BigRoy
Copy link
Member Author

BigRoy commented Feb 5, 2020

This looks exactly like what I described. Can I start using this? ❤️ PR to discuss?

@mkolar were you looking for this too?

@davidlatwe
Copy link
Contributor

Great ! Let me tweak a little bit and will submit a PR for having a look.

@mkolar
Copy link
Member

mkolar commented Feb 6, 2020

Boom. Quicker people lead the world...

Yes this looks exactly like what we planned to try. We'll have to port it to pyblish-lite, but the principle looks sound.

The only difference is in naming. We though pre-collect and collect, David did collect and post-collect. Don't mind either way.

@davidlatwe davidlatwe linked a pull request Feb 6, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants