Consolidate All Project Resources to Compressed Zip File

[RodneyBaker]

This Feature request is primarily for research purposes as there may be several approaches to accomplishing the underlying goal of maintaining a project's resources in one location.

This feature relates to issues users face locating and maintaining all resources associated with any given project. It roughly relates to several other issues identified and/or already addressed to inluce one currently being worked; (#373 Feature request - The ability to add a custom project root through the interface.

Issue:
Resources associated with a Project may be located in a variety of places and if/when the resources move from those locations or the links/pointers change the resource is missing from the Project. In some cases the resource still exists (i.e. it wasn't deleted) but simply missing. Any missing resource however may make the project unusable.

Possible Solution: (Complex)
Using Subversion (SVN) in project workflow/management s likely an ideal solution because of the non-destructive nature of versioning files. At any time a file that i altered can be recovered to an earlier state. The issue with this solution is that considerable setup and maintenance is required to ensure files are properly processed. Additionally, knowledge of versioning processes is required as well installation of supporting software (example: Tortoise SVN, etc.) Another negative to this approach is that the average user of OpenToonz will likely not use versioning until/unless more tightly interwoven into the UI and workflow of even the simplest of projects.

Possible Solution: (Simple?)
Saving all associated files into a compressed file (ex: Zip) with full paths stored will allow for all resources to be stored in one file with the capability of restoring files to original locations. The file's extension could be .zip although naming the extension something otherwise would not change the type of file that it is. For instance, if the file extension is .otz OpenToonz could recognize that for the compressed file that it is and handle it accordingly.

Alternatives:
There are several alternative approaches that may be superior to that of using a standard compressed file such as .zip to collect data. This is especially true given that containers for image files/sequences are being produced to take advantage of image compression algorithms along with the use of advanced codecs. MXF is one such container of which recently .MOX was fielded (by Brendon Bolles) for use in production. NOX/MOX currently has plugins for Adobe After Effects and Premiere/Media Encoder. The idea here is to store the image sequences along with other relevant files for easy access/playback.

Although there is a bigger picture to all of this, this feature request is specifically focused on collecting all resources associated with a project into a single compressed zip file. Consolidation by the user would most likely take place at the end of production where nothing is expected to change. After saving the project (via Save All) the user would then select Consolidate Project which would add all resources into the .zip file at the location specified. This consolidated file would then be stored for archival purposes, kept with the main project file itself, or sent to authorized associates/clients (as applicable).

Thanks! (>:o) )

[konero]

Since OT is a compositor, a self contained file would be bad. It is meant for multi-user and cross-application workflows that composites resources from different locations, for this reason a self contained file can't work. I use TVPaint which uses a self contained file and it is a pain to work with in multi-user environments, I've also had them become corrupted and grow to some incredible sizes.

You can put all your resources in the same directory and zip it manually, how is that any different to a self contained file? Here is how it is setup in my studio. O is for working storage and R is for fast rendering.

If you were to misplace the +drawings folder, either place it back where it used to be or change the path in project settings.

[RodneyBaker]

Thanks Konero, I'll be investigating my workflow to see how I can best optimize my approach.

You said:

If you were to misplace the +drawings folder, either place it back where it used to be or change the path in project settings.

The issue there is that we'd already have to know where the contents of the drawing folder is and have access to it. I can see several situations where that might not be the case or we might think we know where the files are but for one reason or another they simply aren't there (i.e. drive has been removed or folder has been renamed).

Another scenario might be that you are asked to share all of the files of the project with someone else. I'm curious how the average user would do this, especially if they aren't as organized. The just get "Send me all the files please." Would they crack open the Project settings and write down on paper where everything is kept, then go to those locations and copy the files to a new folder in anticipation of compressing the folder in order to send it to the requestor?

My own methodology -at present- is to Save As under a new Scene name file followed up by a Save All which (hopefully) puts everything somewhere near where everything needs to be for easy access. I've discovered at times this isn't always the case but that may relate to using different nightlies as well as the portable indiscriminately.

I do like your approach to having an R drive for rendering because collecting all the resources from the Project usually won't require grabbing all the rendered output as well. This is similar to my use of a folder named Renderfolder dropped wherever I deem necessary. I also have an R drive but at present don't use it much. I would expect to use it for larger productions.

It may be that I'm only just now warming up to the (important!) approach to naming a Project first before launching into creating things. I agree that manual zipping of files already neatly organized is a pretty simple thing to do.

I'll stare at this one some more and also investigate how direct use of SVN via OT might also assist. I started to investigate that but then got distracted by other things.

Thanks for your insight.

[Rayek]

@RodneyBaker I would suggest that you learn how to work with projects in OT, as well as properly saving & naming each level file. Relying on the Save All option is just asking for trouble - your drawings folder will be clogged up with randomly named levels, making it very hard to keep track of things.

Organize all your assets in sub-folders in the relevant project folders before importing them into your animation. This way it is simple to right-mouse click your project folder, and create a zipped version.

And it is also possible to customize the project folder names, and add additional folders automatically. I also set up custom default folders for rendering which I point at a large HDD outside my project folder to prevent it from clogging up with very large exported movies.

The manual states:

Besides the basic default folders you can also add your own default folders, and use the related aliases, by defining a TXT file named project_folders.txt in the Toonz 7.0 stuff\profiles folder. In this file you can also change the order of the folders already available to change the way they are listed in the project settings dialog.

BTW in my opinion Never Ever Work With Absolutely Linked Project Assets.
Which means that you should be careful to place all project files relatively to your project root (all your files should reside in the project folder, with none living on your desktop, or something).

[konero]

@RodneyBaker

Would they crack open the Project settings and write down on paper where everything is kept, then go to those locations and copy the files to a new folder in anticipation of compressing the folder in order to send it to the requestor?

The average user would likely store everything in the same directory. They can literally just Right-click > Send to > Compressed (zipped) folder under Windows.

My own methodology -at present- is to Save As under a new Scene name file followed up by a Save All which (hopefully) puts everything somewhere near where everything needs to be for easy access. I've discovered at times this isn't always the case but that may relate to using different nightlies as well as the portable indiscriminately.

I create a new project first or load an existing one, make sure my file paths are set and make sure to enable append scenepath, then create a new scene and save it using a proper naming convention. Toonz will then automatically organize the drawings folder into folders matching the scenepath including the scene filename.

For example if I saved a scene as \scenes\sa_sc001.tnz then when I save a level Toonz will put it into \drawings\sa_sc001\A.tlv. If I saved a scene as \scenes\seq_01\sc_001.tnz then Toonz will organize the drawings folder as \drawings\seq_01\sc_001\A.tlv.

[Rayek]

@RodneyBaker I learned today that project folders in OpenTOonz may link to any drive and folder on a system - except for the scenes folder. Scenes should always be saved in the project root's scenes folder to keep the file paths integrity intact. The project settings are maintained and kept in the project root in the Stuff folder as well.

This means it is very much possible to have project files all over the place, and I would agree with you that some kind of "package" option, akin to InDesign for example, would be truly useful to have. This option would also have to offer options to exclude certain folders from inclusion in a packaged folder and/or zip. Also handy would be an option to only include all the files that are actually in use by any scene in the project (for cleanup and sharing).

I imagine that unpacking such a folder would create a singular project folder with all the asset folders, settings, and relative paths set up in the project's xml file - just copy to your stuff's project root, and presto: a working project clone.

[RodneyBaker]

Thanks @konero and @Rayek and all others who have been looking hard at the way project resources are stored and managed in OpenToonz. I am trying my best to learn the OpenToonz way of doing thing versus what I tend to use in other programs. I have used Subversion (SVN) as well as Consolidation (to zip file) in another program and that is very likely influencing my desire for such here as well.

@konero said:

Since OT is a compositor, a self contained file would be bad.
and @Rayek said:
Relying on the Save All option is just asking for trouble

We may need to examine this more because with the advent of that 'Save All' option I fairly confident the vast majority of OpenToonz users are going to be using a workflow that includes using 'Save All'. I've been following the initial issues that folks had with saving all resources individually (or losing them!) which led to the addition of 'Save All' as a remedy and popular addition. Adding the 'Save All' option made OpenToonz accessible for many people, especially those who don't have a dedicated project management person/team. Users are certainly going to expect 'Save All' to save all their project's assets.

@Rayek also said:

Scenes should always be saved in the project root's scenes folder to keep the file paths integrity intact.

If scenes should always be saved in the project root folder then that should be easy to do automatically (via Save All, etc.). I must assume that is what Save All currently does. It certainly appears to do that.

@Rayek then elaborated:

I learned today that project folders in OpenTOonz may link to any drive and folder on a system - except for the scenes folder. Scenes should always be saved in the project root's scenes folder to keep the file paths integrity intact. The project settings are maintained and kept in the project root in the Stuff folder as well.

I was following most of the conversation between Shun and Turtletooth regarding project file setup and that's part of the impetus for me requesting the consolidate feature. I need to do my due diligence and actually follow Shun's recommendations though so I can clearly see how OpenToonz is designed to be used as opposed to how I might imagine it to be.

A side point might be that I perceive that users don't always want to setup a project. Sometimes we just want to explore OpenToonz and draw/animate. After exploring however, we very likely will still want to save our results, share them with others and perhaps even refer back them later.

Consolidation is one way of reducing many files to one file without the loss of any information.

I am going to rename this feature request as Feature Request: Consolidate All Project Resources to Compressed Zip File. While the detail of how that compression gets done could be debated if all files referenced by a Project could be added to a zip file with that project's name (i.e. Project.zip) that would resolve this request. If another format is deemed to be more appropriate or optimal than .zip (I get the sense that a format like MXF, MOX or one of the emerging standards might be) then that could be another PR.

Apologies for all of the words.
Thanks everyone for bearing with me. :)

[blurymind]

I suggested something akin to what @konero has in mind here:
#269

This seems to be a duplicate feature request. I am happy that it is getting attention again. :)

[AyameLee]

I like the Maya index @konero and @blurymind mentioned in #373 which I think solves this issue, this would allow zipping, loading, sharing, networking and absolute paths.

[Jose-Moreno]

@RodneyBaker I think this is a great Idea to enhance the Collect Assets feature which does already exist inside OpenToonz.

It is difficult to locate but right now this feature works similar to After Effects' "collect files" which gathers all the footage scattered around in different folders and arranges them into the aliased sub-folders inside your current project folder.

Basically you are using the TNZ file as a beacon to collect the files. For this to work you always have to work by defining OT project root folders. However If you somehow managed to do it wrong like I did the first time it has a super easy fix.

My problem was that I created a small animation test without defining a project root, so everything, levels, meshes, tnz's, hooks, etc were created inside the original folder without a proper structure where I saved the scene file, and since everything had absolute paths I couldn't move them without having to manually change the level settings path inside the file browser dialogue, which by the way is a nightmare.

However the software acknowledged that the root folder was the sandbox, because that's the default if you don't specify anything else, so using the collect files feature was just a matter of importing the scene into a new project and ask OT to collect the files to the new project folder. That was it.

The only thing that wasn't properly "collected" from one project root to another was the output folder because everytime I exported something it was already putting those on the sandbox folder by default, so there was "nothing" to collect in terms of how the routine works. Also, it happily turned my absolute paths into relative ones due to the footage getting aliased inside the new project, so problem solved.

This is from the TOONZ manual, but this feature is working in OpenToonz 1.1.2:
To collect the assets of one or several scenes:

1. Select the scenes for which you want to collect assets.
2. Right-click the selection and choose Collect Assets from the menu
   that opens: all the scene files that were located in external folders are
   copied into the default folders of the project, and the related paths used
   in the scene files are updated.

Hope this helps give perspective for the developers so they don't reinvent the wheel but rather improve the existing feature so we can also ZIP the Collected Projects in one fell swoop 😄

Edit: Added screenshot to facilitate finding the feature

[RodneyBaker]

@Jose-Moreno Thank you! I did not know about the Collect Assets feature (or if someone else told me it went right over my head.

Very useful.

I'll note for anyone looking in that the Collect Assets feature is accessed by the Browser (either shift clicking or Ctrl clicking the scenes desired then Right clicking and selecting Collect Assets. That may or may not be obvious but I'll foot stomp that here.

it happily turned my absolute paths into relative ones

NIce. I like relative paths and was hoping to find ways to get that.
I'll investigate!

[artisteacher]

Collecting assets isn't working for me with OS X Sierra. Fortunately, new external assets can be set to import automatically in the recent nightly builds.

[ghost]

Is that working ok for you @artisteacher ?

[artisteacher]

It's working beautifully! Thanks @turtleTooth

[RodneyBaker]

I should state that using 'Collect Assets' prior to exporting the scene is not needed and can lead to missing files.
The better and simpler process in my experience is to just use the Export Scene option by itself.

Steps:

1. Via Browser or other location where a .tnz scene file can be Right Clicked, Right Click and choose Export Scene.

2. In the Export Scene dialogue near the bottom enter a name for the new project in the Create New Project name field. This will create a new directory in stuff/projects with all the contents of the current scene:
   

3. Navigate to the stuff/projects diretory and zip/compress the project directory

4. Store, archive and/or share the zip file as desired.

[RodneyBaker]

Back some time ago I did research gaining a capability to zip up project directories via Opentoonz but didn't find a good solution that was cross platform short of using a third party library. There are several fairly straightforward solutions for Windows only but that results in a .tar file and not .zip which for some might not be ideal. Interestingly the .tar file can be given the .zip extension and work just fine in most cases but that also is not ideal.

[RodneyBaker]

Tis is a feature that is reasonable to target for v1.8.
We do need to research the best approach given the number of options available.
This especially as it might require the use of a dedicaeted library for the purpose of compression.
Given that much progress has been made recently in larger scale compression (as demonstrated by zstd) that might be a deeper solution and perhaps one that is complemented by a smaller solution for simple projects stored in zip files.

Given that miniz is a library associated with tinyexr that library might be a reasonable way forward but I think even with tinyexr that library is designed to be swapped out for others.

All this to suggest research is still required to see this feature implemented as the original request (by me) is rather trivial in nature compared to what additional features and optimizations can be achieved.