Adding path validator for non-maya nodes

[moonyuet]

Brief description

Adding a path validator for filepaths from non-maya nodes, which are created by plugins such as Renderman, Yeti and abcImport.

Description

As File Path Editor cannot catch the wrong filenpaths from non-maya nodes such as AlembicNodes, It is neccessary to have a new validator to ensure the existence of the filepaths from the nodes.

Additional info

There are some possibilities that some non-maya nodes with filepath attributes missing in the validator.
Please suggest if you catch one or more missing.

Testing notes:

1. Launch Maya from Openpype
2. Create non-maya nodes from the plugins(like importing abc)
3. Run the publisher
4. It will error out and show that the nodes with wrong filepath if it does not exist.

[ynbot]

Task linked: OP-4651 paths validator

[m-u-r-p-h-y]

we should create some easy way to add new nodes with path attributes to the validation as we will miss some or there will be new nodes coming with every renderer in maya.

[m-u-r-p-h-y]

Redshift nodes with file path attribute

RedshiftSprite.tex0
RedshiftBokeh.dofBokehImage
RedshiftLensDistortion.LDimage
RedshiftCameraMap.tex0
RedshiftEnvironment.tex0
RedshiftEnvironment.tex1
RedshiftEnvironment.tex2
RedshiftEnvironment.tex3
RedshiftEnvironment.tex4
RedshiftDomeLight.tex0
RedshiftDomeLight.tex1
RedshiftIESLight.profile
RedshiftLightGobo.tex0
RedshiftNormalMap.tex0
RedshiftProxyMesh.fileName
RedshiftVolumeShape.fileName

[m-u-r-p-h-y]

Vray nodes with file path attribute

VRayTexGLSL.fileName
VRayMtlGLSL.fileName
VRayVRmatMtl.fileName
VRayPtex.ptexFile
VRayLightIESShape.iesFile
VRayMesh.fileName2
VRayMesh.overrideFileName
VRayMesh.alembicLayersFileName
VRayMesh.materialAssignmentsFile
VRayMtlOSL.fileName
VRayTexOSL.fileName
VRayTexOCIO.ocioConfigFile
VRaySettingsNode.causticsFile2
VRaySettingsNode.causticsAutoSaveFile2
VRaySettingsNode.lc_fileName
VRaySettingsNode.lc_autoSaveFile
VRaySettingsNode.imap_fileName2
VRaySettingsNode.imap_autoSaveFile2
VRaySettingsNode.pmap_file2
VRaySettingsNode.pmap_autoSaveFile2
VRayScannedMtl.file
VRayScene.FilePath
VRayScene.parameterOverrideFilePath
VRayMtlMDL.filename
VRayProxy.fileName
VRaySimbiont.file

[moonyuet]

we should create some easy way to add new nodes with path attributes to the validation as we will miss some or there will be new nodes coming with every renderer in maya.

Yes I agree I think it would be great to let the user just add nodes and the attributes in the openpype settings. I will add some default examples for references in the next update PR

[moonyuet]

I have updated the options to allow users adding filename attributes and the related nodes in the setting. Like what you said, it is easier option to deal with the issue as the list of nodes with filename attributes are way too long.

I will set the attributes below as default settings for the user references, it wont affect the validator works if the nodes do not exist.

[antirotor]

I agree with the solution, but lets add the nodes we know (and @m-u-r-p-h-y provided) as default settings so they are already present... No harm in doing that and it can save a lot of time configuring.

[moonyuet]

VRaySimbiont

I have added all the file attributes from the list @m-u-r-p-h-y provided into the settings in the updated PR.

[BigRoy]

Nice stuff - I have added some comments.

Another thing that stood out to me is that it doesn't explicitly tell the user which attributes failed. I think we might want to self.log.error the node attribute with the invalid value:

File does not exist: node.attr = "/path/to/file.png"

Should we log the specific attributes involved?

[BigRoy]

No need to check if targets. If it's an empty targets list the for loop already wouldn't do anything.

[BigRoy]

I'd try to push for less indentation and change this to:

if not file_attr:
    return []

[BigRoy]

Since we're formatting the string in the for loop anyway I see no reason why we need to prefix attr here already.

[BigRoy]

Suggested change

	filepath = cmds.getAttr(target + path_attr)
	filepath = cmds.getAttr("{}.{}".format(target, attr))

[BigRoy]

This change shouldn't be here.

[BigRoy]

I believe this is the wrong settings schema for this plugin. Instead it should be like the other plugins where it can also have optional, enabled , etc. for the ValidatePathForPlugin key.

As such, the attributes it should validate should be a subkey, like project_settings/maya/publish/ValidatePathForPlugin/attributes

@antirotor - thoughts?

[antirotor]

sure, something like:

{
            "type": "dict",
            "collapsible": true,
            "key": "ValidatePathForPlugin",
            "label": "Non-existent Paths in Non-Maya Nodes",
            "children": [
            {
                "type": "boolean",
                "key": "enabled",
                "label": "Enabled"
            },
            {
                "type": "boolean",
                "key": "optional",
                "label": "Optional"
            },
            {
                "type": "boolean",
                "key": "active",
                "label": "Active"
            },
            {
                "type": "dict-modifiable",
                "store_as_list": true,
                "key": "attributes",
                "label": "Attributes",
                "use_label_wrap": true,
                "object_type": {
                    "type": "text"
                }
         }
}

[m-u-r-p-h-y]

OpenVDB

dlOpenVDBShape.filename

GPU Cache

<shape>.cacheFileName

Yeti

pgYetiMayaShape.cfn
pgYetiMayaShape.ofn
pgYetiMayaShape.labf
pgYetiMayaShape.trf
pgYetiMayaShape.liveABCFilename

[BigRoy]

Note

Be aware that the logic involved is relatively rudimentary but the filepaths involved for some of these attributes can be quite different. E.g. some might support <UDIM>, #### (frame padding?), %04d (frame padding, yeti?).

Then additionally if these would involved checking sequences - should it invalidate if ANY frame is missing inside the frame range? Or should we just check to see whether there's any image on disk in that range at all? If we were to check the full range of the sequence against e.g. instance.data["frameStartHandle" and instance.data["frameEndHandle"] then note that some of these attributes might also be influenced by additional attributes that could offset a sequence, or in some cases even have a custom inputTime attribute driving the node to what frames it would sample.

Currently if the logic here would check an attribute that has a dynamic file pattern like a sequence or a UDIM texture it would invalidate the publishing.

[moonyuet]

Note

Be aware that the logic involved is relatively rudimentary but the filepaths involved for some of these attributes can be quite different. E.g. some might support <UDIM>, #### (frame padding?), %04d (frame padding, yeti?).

Then additionally if these would involved checking sequences - should it invalidate if ANY frame is missing inside the frame range? Or should we just check to see whether there's any image on disk in that range at all? If we were to check the full range of the sequence against e.g. instance.data["frameStartHandle" and instance.data["frameEndHandle"] then note that some of these attributes might also be influenced by additional attributes that could offset a sequence, or in some cases even have a custom inputTime attribute driving the node to what frames it would sample.

Currently if the logic here would check an attribute that has a dynamic file pattern like a sequence or a UDIM texture it would invalidate the publishing.

I think we can try to use clique library to query those files to see if possible to validate the files, but definitely needs time to write and test something out.

[m-u-r-p-h-y]

Again we should first provide the feature and improve it over time (not waiting to be perfect from the first go)
🙏
adding support to all tokens <####> <%04d> <F> <f> <UDIM> <udim> <frame> can be another PR

[moonyuet]

Again we should first provide the feature and improve it over time (not waiting to be perfect from the first go) 🙏 adding support to all tokens <####> <%04d> <F> <f> <UDIM> <udim> <frame> can be another PR

I agree we can provide the feature first, while we can work on validating the tokens in the next PR, as it needs some time to test and see if it works as expected.

[tokejepsen]

Missing indentation for attributes entries.

[tokejepsen]

Less indentation than normal?

[antirotor]

Apart of the guard statement that is in this case more of cosmetic issue, it looks good.

[antirotor]

A would use here the guard statement to ease up on nesting and making the code more readable:

Suggested change

	file_attr = validate_path["ValidatePathForPlugin"]["attribute"]
	if file_attr:
	# get the nodes and file attributes
	for node, attr in file_attr.items():
	# check the related nodes
	targets = cmds.ls(type=node)
	for target in targets:
	# get the filepath
	file_attr = "{}.{}".format(target, attr)
	filepath = cmds.getAttr(file_attr)
	if filepath and not os.path.exists(filepath):
	self.log.error("File {0} not exists".format(filepath)) # noqa
	invalid.append(target)
	file_attr = validate_path["ValidatePathForPlugin"]["attribute"]
	if not file_attr:
	return invalid
	# get the nodes and file attributes
	for node, attr in file_attr.items():
	# check the related nodes
	targets = cmds.ls(type=node)
	for target in targets:
	# get the filepath
	file_attr = "{}.{}".format(target, attr)
	filepath = cmds.getAttr(file_attr)
	if filepath and not os.path.exists(filepath):
	self.log.error("File {0} not exists".format(filepath)) # noqa
	invalid.append(target)

[moonyuet]

Apart of the guard statement that is in this case more of cosmetic issue, it looks good.

I have updated the PR with the fix on the cosmetic issue. Thx!

[BigRoy]

This PR has broken maya publishing because THIS ValidatePathForPlugin attribute does not exist in settings. It is ValidatePluginPathAttributes instead.