Feature/PYPE-611 build workfile

[iLLiCiTiT]

PR in pype-config: https://bitbucket.org/pypeclub/pype-config/pull-requests/149/draft-pype-611-build-workfile
Implemented few methods that can help to build first workfile.

1.) Create first workfile:

• not yet complete!
• need discussion about what to happend when already "first" workfile exists
  • override (then what to do if exists 2nd version?)
  • do nothing
  • crash and handle that in host (show message or something)
  • ...

2.) Load containers for first workfile:

• it is possible to specify what to load for first workfile
  • specifications are only per host at first place (maya, nuke, ...)
  • each specification is for task names (compositing, modeling,... )
    • only one specification for one task name (if task name is in more than one specification then first found is used)
    • but it is possible to have same specification for multiple task names
  • specification define subset filters, representation filters and loader names
    • subset filters:
      • family - main family of published subset
      • subset_name_filters - regexes of subset name OPTIONAL skipped if not set
    • subset filters are strict both family and regex must match (at least one combination)
    • representation names and loaders loop looks for first available combination that can be used to load container
      • loop progressively traverses through each loaders for each representation, so order of representation names and loaders matters
• it is possible to use linked assets (Not yet implemented logic for that. Method get_asset_link returns empty list)
• whole loading method returns all loaded containers -> list of dictionaries, each dictionary has asset_entity and containers
  • at the time of PR list may have one or none items (due to missing asset link implementation)
  • containers are objects returned by loader

[mkolar]

This is working really well !!!

I've tried with multiple configurations on multiple tasks and it got what I expected every time. Well done!

So let's just deal with your original points to discuss and it will be ready to merge.

[mkolar]

1.) Create first workfile:

Let's drop the workfile creation altogether. I think this should simply load all the expected content into the current file and let user deal with the actual saving. I can very much imagine using this on later version of the file, if I need to reset for example.

specifications are only per host at first place (maya, nuke, ...)

that's fine, but let's move the preset from presets/plugins/host/workfile_build.json to presets/host/workfile_build.json it doesn't involve any particular plugin or category, hence should be in the general host area.

[BigRoy]

Not entirely sure what this PR is intended to do - admittedly the code was a bit hard to read through and get a glimpse of what exactly it is trying to do. Anyway, did have some notes on some of the code that stood out as I skimmed the code.

Also, this could really benefit from a README.md or something to explain how to use it. Hehe.

[BigRoy]

This comment could be improved by describing why it's breaking there as opposed to explaining that it's just breaking the loop. Now it seems to explain what the break command does. Should it maybe instead explain e.g. # break variants loop on finding the first matching variant?

[mkolar]

There's also been a conversation about turning this into it's own class (which i agree with btw), but unfortunately, it fell in the battle of converting repos to public...meaning I accidentaly deleted the convo :).

@antirotor can you please have a look at the code one more time and comment? We need to keep in mind that this will likely be heavily expanded in the future with things like linking assets together, loaded asset packages and possibly triggering things like publishing (auto scene build and send to farm anyone?) like what was discussed here getavalon/core#130

@jezscha I've tested this in nuke as well and it works really well. Can you please have a look? Is there anything, in particular, you need it to return to be able to hook to it the kind of scene build we have in the nuke now? As far as I can tell you can do as you please with the loaded containers, considering the function returns them after it runs.

[antirotor]

I have few remarks about code structure:

1. I would get it out of lib.py. This deserves its own package. It is quite messy to track changes in file with +500 lines of code and this one has ~1300.

2. There is no ftrack or host specific code so maybe it is not necessary to have it as a class. But I would nevertheless mark functions as private - and I believe most of them are. If there is even slightest chance that this will have to use host specific code, I would wrap it as a class so each host specific implementation can inherit and override whats necessary without need to import specific functions from some package.

3. I would break load_containers_by_asset_data() into more pieces. Linter warning about to high complexity is in this case very true - consider writing test for such function. It would be impossible to test all lines of this complex function. Most of it are for loops that can be simplified or made more readable by using filter functions, generators, etc.

4. More docstrings 🙏

5. Some value pairs used through the code could be written as tuples rather then independent pair of strings as they are connected. For example:

   variant_loader_names = variant["loaders"]
   variant_loader_count = len(variant_loader_names)

   can be:

   var_loader = (variant["loaders"], len(variant_loader_names))

   as Tuple is immutable it ensures those two values belongs together and using it later take less
   code writing (accessing them like var_loader[0] is shorter then variant_loader_count. Yes, that is
   cosmetics 😺

6. If that code would be taken out into separate class or package it is good practice to define all regexes somewhere at the top. And when used inside loops it's best to compile them first (re.compile)

And one last thing - I sort of dislike name variant. I think that in this case it isn't something variable per se maybe something like data template or subset template to be more correct.

[antirotor]

Suggested change

	subset = valid_subsets_by_id[subset_id]
	subset_name = subset["name"]
	variant = variants_per_subset_id[subset_id]
	variant_loader_names = variant["loaders"]
	variant_loader_count = len(variant_loader_names)
	variant_repre_names = variant["repre_names"]
	variant_repre_count = len(variant_repre_names)
	subset = (valid_subsets_by_id[subset_id], subset["name"])
	variant = variants_per_subset_id[subset_id]
	variant_loader = (variant["loaders"], len(variant_loader_names))
	variant_repre = (variant["repre_names"], len(variant_repre_names))

[antirotor]

see my review comment about using tuples

[antirotor]

re-submitted review text by accident, please ignore.

[iLLiCiTiT]

Current state of this PR was "for testing" if it works because I wasn't sure about fully understanding of the feature and my implementation, so I'm glad most of the comments are not logic based :)

• @BigRoy good spots. (very good)
• @antirotor agree with all points but we'll need to define few thing for points 1, 2, 3. Not just for this feature but all similar features.

[iLLiCiTiT]

I realized I didn't understood the feature properly:
Whole part about Build "First" Workfile doesn't make sence. It should be Build Workfile or similar, you can came up with proper name.

Both methods create_first_workfile and get_last_workfile_path are not required for this feature, even more they must not be in this feature. Work file should be build by user through Workfiles tool. Am I right?

[mkolar]

Both methods create_first_workfile and get_last_workfile_path are not required for this feature, even more they must not be in this feature. Work file should be build by user through Workfiles tool. Am I right?

yes exactly. even though I can very much imagine that extending this by actually saving the workfile will be desirable in some cases. This framework should focus on finding, and loading required content only.

For example to recreate the behavior of current build first workfile in nuke, that is used in production, we'll need to tweak the function inside pype.nuke to use this for the actual loading of representation, but then taking the containers and organising them into backdrops and whatnot. Nuke specific build_workfile function must be responsible for that post processing though.

[mkolar]

And one last thing - I sort of dislike name variant. I think that in this case it isn't something variable per se maybe something like data template or subset template to be more correct.

fully agreed. We should come up with name that we can use across pype for these types of thing. It's the same type of preset as is used for extract_review and will be used for extract_burnins for example. Currenly we call those individual presets profiles. So when you se h264 and edit quicktimes with different settings and burnins and whatever, they are consider as two profiles.

Might be a good candidate for a name here as well?

"Create a new scene building profile where you specify what task it operates on, what subsets and representations you are interested in and what loader should be used to bring them into the scene"

sounds pretty reasonable to me

[iLLiCiTiT]

Work file should be build by user through Workfiles tool.

It is because I'm not using same method as uses Workfiles tool. That is because there's not such a method in avalon.api, workfiles tool has it's own methods for that. So when way of creation of workfiles changes in Workile tool, method implemented in other place will break production. Which will probably happen in nuke with multiroots BTW.