Implement Application discovery and launch in core

[BigRoy]

Issue

Currently the Launcher allows to launch applications within the context of a project with its completely contained logic - this should be moved to the Avalon core where applications should be able to be listed, initialized and launched.

The Launcher currently uses .toml files to describe applications as described here: https://getavalon.github.io/2.0/reference/#project-executable-api

The api in core would return Application classes that can run or get triggered within a specific Session, e.g. like Maya.

---

A proposed api by @mottosso can be found here

from avalon import api

for app in api.discover(api.Application):
  print(app.name)

Other references:

• ftrack actions and frack actions code sample
• shotgun actions and shotgun maya actions code

---

Use cases

Some of my use cases:

• Launch maya for a given task (similar to what launcher does now)
• Initialize the application directory for many tasks (through code) without launching the application, e.g. set up many Maya folders.
• Launch project manager or other apps that are not necessarily inside asset or task; e.g. a project manager could create a new project that does not yet have any assets/tasks.
• List what applications I can run in my currently active environment or session; e.g. in a project outside of silo/asset/task, what Application/actions can I run?

Eventually be able to list the currently available Applications (or Actions?) in a current context, similar to ftrack, shotgun or even the current launcher. Here's a visual from ftrack:

[BigRoy]

Here's a proposal from my end.

1. Applications (or just "Actions"?) can be implemented in Python similar to Loaders and Creators and are discovered a in similar manner.
2. The Action itself can define whether or not it's visible within a certain context or session - as such it would need to implement a method that would return its compatibility/availability. For example is_compatible
3. It should still become possible to "initialize" an application environment without actually launching it. Maybe every action can als hold a "initialize()" method. Or maybe it's a matter of running Action.process with a keyword argument, like Action.process(launch=False)

class Action(object):
    """Run an action within a context"""

    name = None
    label = None
    icon = None
    order = 0

    def launch(self, **kwargs):
         pass

    def is_compatible(self, session):
        """Return whether this action can be triggered in the specific session"""
        pass

A Maya application could then run like:

# pseudocode
class Maya2018(Action):
    name = "maya2018"
    label = "Maya 2018"
    icon = "path/to/maya/icon.png"

    def is_compatible(self, session):
        required = ["AVALON_ASSET", "AVALON_SILO", "AVALON_TASK", "AVALON_PROJECT", "AVALON_PROJECTS", "AVALON_WORKDIR"]
        missing = [x for x in required if x not in session]
        if missing:
            return False
        return True        

    def process(self, **kwargs):
        env = self.env.copy()
        env["PYTHONPATH"] = "xx/yy"

        if kwargs.get("launch", True):
            return lib.launch("maya2018")

For backwards compatibility and usage of the .toml files a function could be written that would create a class for each .toml and register that.

The Project manager could then run like:

# pseudocode
class ProjectManagerAction(Action):
    def is_compatible(self, session):
        # This can run anywhere?
        return True

    def process(self, **kwargs):
        import avalon.tools.projectmanager.app as app
        app.show()

Then to show the available actions in your current context:

actions = api.discover(api.Action)
session = api.Session.copy()
available = api.actions_by_session(actions, session)
print available

Or to check compatibility for a single one.

# Note: this code assumes the discover return action classes - not instances. 
#   and it also assumes the "is_compatible" method is a classmethod or 
#   staticmethod so it can run prior to initialization of the action.
Action = next(api.discover(api.Action))
if Action.is_compatible(session):
    action = Action(session)
    action.process()

[BigRoy]

Here's some code that functionally works similar to the current launcher.

@lib.log
class Action(object):
    """A custom action available"""
    name = None
    label = None
    icon = None
    order = None

    def is_compatible(self, session):
        """Return whether the class is compatible with the Session."""
        return True

    def process(self, session, **kwargs):
        pass


class Application(Action):
    """Default application launcher

    This is a convenience application Action that when "config" refers to a
    loaded application `.toml` this can launch the application.

    """

    config = None

    def is_compatible(self, session):
        required = ["AVALON_PROJECTS",
                    "AVALON_PROJECT",
                    "AVALON_SILO",
                    "AVALON_ASSET",
                    "AVALON_TASK"]
        missing = [x for x in required if x not in session]
        if missing:
            self.log.debug("Missing keys: %s" % (missing,))
            return False
        return True

    def process(self, session, **kwargs):

        session = session.copy()
        session["AVALON_APP"] = self.config["application_dir"]
        session["AVALON_APP_NAME"] = self.name

        # Compute work directory
        project = io.find_one({"type": "project"})
        template = project["config"]["template"]["work"]
        workdir = _format_work_template(template, session)
        session["AVALON_WORKDIR"] = workdir

        # Build environment
        env = os.environ.copy()
        env.update(self.config.get("environment", {}))
        env.update(session)

        # Create working directory
        workdir_existed = os.path.exists(workdir)
        if not workdir_existed:
            os.makedirs(workdir)
            self.log.info("Creating working directory '%s'" % workdir)

            # Create default directories from app configuration
            default_dirs = self.config.get("default_dirs", [])
            if default_dirs:
                self.log.debug("Creating default directories..")
                for dirname in default_dirs:
                    try:
                        os.makedirs(os.path.join(workdir, dirname))
                        self.log.debug(" - %s" % dirname)
                    except OSError as e:
                        # An already existing default directory is fine.
                        if e.errno == errno.EEXIST:
                            pass
                        else:
                            raise

        # Perform application copy
        for src, dst in self.config.get("copy", {}).items():
            dst = os.path.join(workdir, dst)

            try:
                self.log.info("Copying %s -> %s" % (src, dst))
                shutil.copy(src, dst)
            except OSError as e:
                self.log.error("Could not copy application file: %s" % e)
                self.log.error(" - %s -> %s" % (src, dst))

        # Launch
        if kwargs.get("launch", True):
            executable = lib.which(self.config["executable"])

            return lib.launch(executable=executable,
                              args=self.config.get("args", []),
                              environment=env,
                              cwd=workdir)

This is then how I'm currently generating the App actions based on a project configuration.

def register_project_apps():

    project = io.find_one({"type": "project"})

    assert project is not None, "This is a bug"

    # Register the applications by `.toml`
    apps = project["config"]["apps"]
    for app in apps:
        try:
            app_definition = lib.get_application(app['name'])
        except Exception as exc:
            print("Unable to load application: %s - %s" % (app['name'], exc))
            continue

        # Define the application class
        class App(pipeline.Application):
            name = app['name']
            label = app.get("label", app["name"])
            config = app_definition.copy()

        api.register_plugin(pipeline.Action, App)

Then to run a specific one:

# List registered apps
for Action in api.discover(pipeline.Action):

    if Action.name == "maya2018":
        action = Action()
        action.process(api.Session.copy())

[BigRoy]

Here's a prototype of the above implemented into the Launcher.

This is the project manager action:

class ProjectManagerAction(avalon.api.Action):
    name = "projectmanager"
    label = "Project Manager"
    icon = "gear"

    def is_compatible(self, session):
        return "AVALON_PROJECT" in session

    def process(self, session, **kwargs):

        import avalon.lib as lib

        return lib.launch(executable="python",
                          args=["-m", "avalon.tools.projectmanager",
                                session['AVALON_PROJECT']])

This will only show when "AVALON_PROJECT" is available in the session - meaning the action is compatible to be processed (or triggered, or launched, or whatever you'd call it).

And this is how the .toml are collected from project configuration:

def get_apps(project):
    """Define dynamic Application classes for project using `.toml` files"""

    import avalon.lib
    import avalon.api as api

    apps = []
    for app in project["config"]["apps"]:
        try:
            app_definition = avalon.lib.get_application(app['name'])
        except Exception as exc:
            print("Unable to load application: %s - %s" % (app['name'], exc))
            continue

        icon = app_definition.get("icon", app.get("icon", "folder-o"))

        action = type("app_%s" % app["name"],
                      (api.Application,),
                      {
                          "name": app['name'],
                          "label": app.get("label", app['name']),
                          "icon": icon,
                          "config": app_definition.copy()
                      })

        apps.append(action)

    return apps

With this change the Launcher practically knows nothing about "launching" an application, it can just trigger actions - of which some happen to launch applications.

[tokejepsen]

Without having touched Avalon yet, this looks pretty good.

It should still become possible to "initialize" an application environment without actually launching it. Maybe every action can als hold a "initialize()" method.

Think separating out the launching code into dedicated methods would give a nicer structure, and less per method responsibilities. Adding to that I would maybe think about an environ method/variable for fetching just the environment for the launch.

Manipulating Action Launch

One thing that has given Ftrack's application actions a lot more flexibility has been the ability to modify the environment and launch arguments; http://ftrack-connect.rtd.ftrack.com/en/latest/developing/hooks/application_launch.html
This enables developers to build on top of existing actions without having to re-implement the discovery and launch code logic.

[BigRoy]

Think separating out the launching code into dedicated methods would give a nicer structure, and less per method responsibilities. Adding to that I would maybe think about an environ method/variable for fetching just the environment for the launch.

Could you elaborate slightly more on this? Perhaps a small psuedocode sample?

This enables developers to build on top of existing actions without having to re-implement the discovery and launch code logic.

Note that the Actions here currently allow **kwargs in process() which could hold any additional that and logic that you could use within the process method.

[tokejepsen]

Could you elaborate slightly more on this? Perhaps a small psuedocode sample?

class Application(Action):
    """Default application launcher

    This is a convenience application Action that when "config" refers to a
    loaded application `.toml` this can launch the application.

    """

    config = None

    def is_compatible(self, session):
        required = ["AVALON_PROJECTS",
                    "AVALON_PROJECT",
                    "AVALON_SILO",
                    "AVALON_ASSET",
                    "AVALON_TASK"]
        missing = [x for x in required if x not in session]
        if missing:
            self.log.debug("Missing keys: %s" % (missing,))
            return False
        return True

    def environ(self, session):
        env = {}
        env["AVALON_APP"] = self.config["application_dir"]
        env["AVALON_APP_NAME"] = self.name

        # Compute work directory
        project = io.find_one({"type": "project"})
        template = project["config"]["template"]["work"]
        workdir = _format_work_template(template, session)
        env["AVALON_WORKDIR"] = workdir

        return env

    def initialize(self, session):

        # Create working directory
        workdir_existed = os.path.exists(workdir)
        if not workdir_existed:
            os.makedirs(workdir)
            self.log.info("Creating working directory '%s'" % workdir)

            # Create default directories from app configuration
            default_dirs = self.config.get("default_dirs", [])
            if default_dirs:
                self.log.debug("Creating default directories..")
                for dirname in default_dirs:
                    try:
                        os.makedirs(os.path.join(workdir, dirname))
                        self.log.debug(" - %s" % dirname)
                    except OSError as e:
                        # An already existing default directory is fine.
                        if e.errno == errno.EEXIST:
                            pass
                        else:
                            raise

        # Perform application copy
        for src, dst in self.config.get("copy", {}).items():
            dst = os.path.join(workdir, dst)

            try:
                self.log.info("Copying %s -> %s" % (src, dst))
                shutil.copy(src, dst)
            except OSError as e:
                self.log.error("Could not copy application file: %s" % e)
                self.log.error(" - %s -> %s" % (src, dst))

    def launch(self, session, **kwargs):

        executable = lib.which(self.config["executable"])

        return lib.launch(executable=executable,
                          args=self.config.get("args", []),
                          environment=self.environ(),
                          cwd=workdir)

Note that the Actions here currently allow **kwargs in process() which could hold any additional that and logic that you could use within the process method

If you are using someone else's Maya action (maybe this is not the intend, then just ignore this), and you want to append an extra flag to the launch without duplicate/modify the action, how would you do that?

[BigRoy]

If you are using someone else's Maya action (maybe this is not the intend, then just ignore this), and you want to append an extra flag to the launch without duplicate/modify the action, how would you do that?

Ah, yes - purely for inheritance. I agree - this would help towards that design. 👍 Thanks.

[mottosso]

project = io.find_one({"type": "project"})

Just don't call the database in this object; treat it as a data container only. If it returns an environment, it should only be the environment it knows; which is the variables set in the config. They can be merged with os.environ or things from the database after the fact.

[mottosso]

Though if it's an "action" then it'd be ok. I think the concept of what we first discussed about an Application object representing the .toml files have vanished from here; which is ok, but better not call it just Application then. Also there is no reference to the .toml file here (is there?) only a reference to the data in the database.

Better call it an application launcher action or the like. Actually, how about calling it Launcher? Like the Creator, Loader plug-ins? That's pretty much what it is. A plug-in for the Launcher.

I still think we need a discovery for .toml files, to get a hold of environment variables, default directories and such. How are you getting those currently?

[BigRoy]

Just don't call the database in this object; treat it as a data container only. If it returns an environment, it should only be the environment it knows; which is the variables set in the config. They can be merged with os.environ or things from the database after the fact.

I've thought about this - but I have no clue how that would work. The application being launched describes the application directory required to format the AVALON_WORKDIR - as such, outside of the action I have no means to set up this AVALON_WORKDIR.

The thing you are right about though, is that it's not explicit enough that the passed session's project is being collected in the io.find_one call, to be more explicit it should temporarily set api.Session['AVALON_PROJECT'] using the one from the passed session object and query the project then.

I think the concept of what we first discussed about an Application object representing the .toml files have vanished from here; which is ok, but better not call it just Application then. Also there is no reference to the .toml file here (is there?) only a reference to the data in the database.

Actually, the config attribute on the Application class is the parsed .toml file. I'm currently dynamically creating these Application instances per .toml like this:

def get_apps(project):
    """Define dynamic Application classes for project using `.toml` files"""

    import avalon.lib
    import avalon.api as api

    apps = []
    for app in project["config"]["apps"]:
        try:
            app_definition = avalon.lib.get_application(app['name'])
        except Exception as exc:
            print("Unable to load application: %s - %s" % (app['name'], exc))
            continue

        icon = app_definition.get("icon", app.get("icon", "folder-o"))
        color = app_definition.get("color", app.get("color", None))

        action = type("app_%s" % app["name"],
                      (api.Application,),
                      {
                          "name": app['name'],
                          "label": app.get("label", app['name']),
                          "icon": icon,
                          "color": color,
                          "config": app_definition.copy()
                      })

        apps.append(action)

    return apps

And the Application class just inherits Action yet implements the previous launching behavior for applications when config is set accordingly. I'll try to push some branches so we can go from there.

The avalon.lib.get_application parses the .toml application.

[tokejepsen]

@BigRoy This looks like its implemented in your fork of the launcher?

[BigRoy]

Yes, I believe so.

[tokejepsen]

Should we try to merge those changes?

[BigRoy]

Yes, great. Could you test it out on your end first? Just to be sure. :)

[BigRoy]

These should be merged with: 5c7f9d6