Skip to content
This repository has been archived by the owner on Jul 21, 2022. It is now read-only.

Ct 1501 add an exclude list to the dependency scanner #304

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Ct 1501 add an exclude list to the dependency scanner #304

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
780e3ec
Adds textures and standins to the ass scraper
Sep 4, 2020
b68f118
Adds a parameter to exclude diving deeper into a given list of paths
Sep 6, 2020
2f9fdeb
Adds a parameter to exclude paths from the depedency scanner
Sep 6, 2020
aabd0d5
Fixes logging call that was missing parameters
Sep 6, 2020
3515650
Forces all dependency paths to be normalized
Sep 7, 2020
d5fc45f
Adds support for glob patterns for leaf/exclude paths
Sep 7, 2020
f9f07ac
Switches to using external file(s) to contain leaf/exclude patterns
Sep 7, 2020
ba47694
Addresses PR comments
Sep 30, 2020
4a58e34
Makes list variables plural
Oct 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 17 additions & 13 deletions conductor/lib/maya_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -472,30 +472,32 @@ def get_render_layers_info():
return render_layers


def collect_dependencies(node_attrs, leaf_path_list=None, exclude_path_list=None):
def collect_dependencies(node_attrs, leaf_path=None, exclude_path=None):
jesseconductor marked this conversation as resolved.
Show resolved Hide resolved
'''
Return a list of filepaths that the current maya scene has dependencies on.
This is achieved by inspecting maya's nodes. Use the node_attrs argument
to pass in a dictionary

leaf_path_list: A list of strings. A list of paths to skip for nested dependencies.
leaf_path: A list of strings. A list of paths to skip for nested dependencies.
Ex: A maya file that itself will be included but not textures, or
other nested references will be.
exclude_path_list: A list of strings. A list of paths to exclude from the
exclude_path: A list of strings. A list of paths to exclude from the
dependency scanner.
'''

assert isinstance(node_attrs, dict), "node_attrs arg must be a dict. Got %s" % type(node_attrs)

leaf_path_list = leaf_path_list or []
exclude_path_list = exclude_path_list or []
leaf_path = leaf_path or []
exclude_path = exclude_path or []

# TODO: Temporary hack to work around renderman 23.3 bug.
# Record the active renderer so that we can restore it after making this cmds.file call
active_renderer = get_active_renderer()
# Note that this command will often times return filepaths with an ending "/" on it for some reason. Strip this out at the end of the function
dependencies = cmds.file(query=True, list=True, withoutCopyNumber=True) or []

# Strip errant dependences (another part of the renderman bug above).
# normpath is used as it's possible (in Windows) to have a path with mixed back/forward slashes
dependencies = [os.path.normpath(path) for path in dependencies if not path.endswith('_<user')]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the normpath?

Copy link
Contributor Author

@jesseconductor jesseconductor Sep 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my tests I found that I could easily end-up with a path that had mixed forward and back slashes (on Windows). I added that as a comment.

logger.debug("maya scene base dependencies: %s", dependencies)
# Reinstate active renderer
Expand Down Expand Up @@ -526,21 +528,23 @@ def collect_dependencies(node_attrs, leaf_path_list=None, exclude_path_list=None
# directory (i.e. it doesn't have any real smarts about path resolution, etc).
# NOTE: that this command will oftentimes return filepaths with an ending "/" on
# it for some reason. Strip this out at the end of the function
#
# normpath() is used as it's possible (in Windows) to have a path with mixed back/forward slashes
path = os.path.normpath(cmds.file(plug_value, expandName=True, query=True, withoutCopyNumber=True))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the normpath?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same answer as above...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, I've tried to narrow the places in our client code for where/when paths are manipulated/conditioned. The majority of the time it occurs after dependency scraping (rather than before, or during). Specifically the process_upload_filepath function is called to explode/resolve/normalize any paths that come out of the dependency scraping phase. See here.
There certainly are exceptions to this (process_upload_filepath gets called several times within the collect_dependencies function), but they occur within isolated logic leaves (not affecting other logic leaves).
I'm not saying that this will break anything. But I wouldn't be surprised if it did.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's needed hear so that it can be compared to values in exlude_paths and leaf_paths

logger.debug("%s: %s", plug_name, path)

if path in exclude_path_list:
if path in exclude_path:
logger.info("Skipping depedency '{}' - in exclusion list".format(path))
continue

if path in leaf_path_list:
if path in leaf_path:
logger.info("Skipping nested scanning of '{}' - in leaf list".format(path))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The log entry message doesn't match the code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you continued'd here, you'd be able to remove the duplicated if path not in leaf_path_list: in this function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No.. because the path itself still needs to be added to the list of dependencies on line 640. I struggled on how to do this cleanly... It really needs a class-based approach. I know @hoolymama did a big refactor on the dependency scanner. Perhaps we can merge that in to take advantage of the refactor?


# ---- XGEN SCRAPING -----
# For xgen files, read the .xgen file and parse out the directory where other dependencies may exist
if node_type == "xgmPalette":

if path not in leaf_path_list:
if path not in leaf_path:
maya_filepath = cmds.file(query=True, sceneName=True)
palette_filepath = os.path.join(os.path.dirname(maya_filepath), plug_value)
xgen_dependencies = scrape_palette_node(node, palette_filepath) + [palette_filepath]
Expand All @@ -553,14 +557,14 @@ def collect_dependencies(node_attrs, leaf_path_list=None, exclude_path_list=None

# ---- VRAY SCRAPING -----
if node_type == "VRayScene":
if path not in leaf_path_list:
if path not in leaf_path:
vrscene_dependencies = parse_vrscene_file(path)
logger.debug("vrscene dependencies: %s" % vrscene_dependencies)
dependencies += vrscene_dependencies

# ---- YETI SCRAPING -----
if node_type == "pgYetiMaya":
if path not in leaf_path_list:
if path not in leaf_path:
yeti_dependencies = scrape_yeti_graph(node)
logger.debug("yeti dependencies: %s" % yeti_dependencies)
dependencies += yeti_dependencies
Expand All @@ -584,7 +588,7 @@ def collect_dependencies(node_attrs, leaf_path_list=None, exclude_path_list=None
# file in an .ass sequence will have the same file dependencies, so don't bother reading every
# ass file. Perhaps dangerous, but we'll cross that bridge later (it's better than reading/loading
# potentially thousands of .ass files)
if path not in leaf_path_list:
if path not in leaf_path:
ass_filepath = file_utils.process_upload_filepath(path, strict=True)[0]
ass_filepaths.append(ass_filepath)

Expand All @@ -597,7 +601,7 @@ def collect_dependencies(node_attrs, leaf_path_list=None, exclude_path_list=None
# will have it's corresponding .rlf file here:
# renderman/ribarchives/SpidermanRibArchiveShape/SpidermanRibArchiveShape.job.rlf
if node_type == "RenderManArchive" and node_attr == "filename":
if path not in leaf_path_list:
if path not in leaf_path:
archive_dependencies = []
rlf_dirpath = os.path.splitext(path)[0]
rlf_filename = "%s.job.rlf" % os.path.basename(rlf_dirpath)
Expand Down Expand Up @@ -626,7 +630,7 @@ def collect_dependencies(node_attrs, leaf_path_list=None, exclude_path_list=None
# to exist on disk). This is not a perfect assumption, but we can adjust as
# needed...perhaps by querying the caching mode.
if node_type == "RedshiftOptions" and file_utils.RX_FRAME_REDSHIFT in path:
if path not in leaf_path_list:
if path not in leaf_path:
logger.debug("Resolving path expression: %s", path)
redshift_filepaths = file_utils.process_upload_filepath(path, strict=False)
if redshift_filepaths:
Expand Down
17 changes: 7 additions & 10 deletions conductor/submitter_maya.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,8 +226,8 @@ class MayaConductorSubmitter(submitter.ConductorSubmitter):

def __init__(self, parent=None):

self.dependency_leaf_glob_file_path = None
self.dependency_exclude_glob_file_path = None
self.dependency_leaf_glob_file_path = os.environ.get('CONDUCTOR_DEPSCAN_LEAF_FILE')
self.dependency_exclude_glob_file_path = os.environ.get('CONDUCTOR_DEPSCAN_EXCLUDE_FILE')

super(MayaConductorSubmitter, self).__init__(parent=parent)
self.setMayaWindow()
Expand Down Expand Up @@ -384,18 +384,15 @@ def collectDependencies(self):
# Get all of the node types and attributes to query for external filepaths on
resources = common.load_resources_file()
dependency_attrs = resources.get("maya_dependency_attrs") or {}

leaf_glob_file_path = os.environ.get('CONDUCTOR_DEPSCAN_LEAF_FILE', self.dependency_leaf_glob_file_path)
exclude_glob_file_path = os.environ.get('CONDUCTOR_DEPSCAN_EXCLUDE_FILE', self.dependency_exclude_glob_file_path)


leaf_path_list = []
exclude_path_list = []

if leaf_glob_file_path is not None:
leaf_path_list = file_utils.expand_paths_from_file(leaf_glob_file_path)
if self.dependency_leaf_glob_file_path is not None:
leaf_path_list = file_utils.expand_paths_from_file(self.dependency_leaf_glob_file_path)

if exclude_glob_file_path is not None:
exclude_path_list = file_utils.expand_paths_from_file(exclude_glob_file_path)
if self.dependency_exclude_glob_file_path is not None:
exclude_path_list = file_utils.expand_paths_from_file(self.dependency_exclude_glob_file_path)

logger.debug("Using expanded leaf file list: {}".format(leaf_path_list))

Expand Down