MLT file is using absolute paths even for resources inside the project folder (bad for portability and projects in version control)

[geekley]

Version: Windows 7 64-bit; Shotcut 19.02.28

Firstly, I must say I really like the choice of making the MLT project file be a XML text file (easy to inspect), instead of some binary format, and have it be created inside a project folder. This is particularly good for projects you intend to put on version control.
I would like this project file to always use relative paths when the resource is inside the project folder, for better portability of projects. You should be able to move the project folder to somewhere else without breaking any path references. External references can remain as absolute paths.

The thing is, I've noticed the shotcut:detail property uses absolute paths (and I can't remove them from the file, they are generated again).

Also, I've somehow ended up with a project file containing absolute file paths on the resource property inside producer, even though the referenced resource file was inside the project folder. I think it was when I fixed missing references to point to the files (now inside the project folder) then it used absoluted paths because the "repaired" project file is saved with a different name.

So, my suggestion to avoid this kind of problem is to display a dialog whenever users import a file located outside the project so that they can have the option to either copy/move it to the project folder (and keep the reference as a relative path) or link to the absolute path anyways. This would make things more explicit for the user and avoid any confusion when moving things around.

[ddennedy]

It is by design that shotcut:detail uses absolute path; that is the detail. This is only used for annotation as a tool tip; it is not used to locate files. Relative paths to media assets on the <property name="resource"> in the XML should be working for files in the project folder. Provide an example MLT XML file with the problem or the steps to circumvent or break this.

[ddennedy]

display a dialog whenever users import a file located outside the project so that they can have the option to either copy/move it to the project folder

I do not plan to use a dialog; rather, there will be options to copy or move in the New Project view, and then it just happens always for that project.

[geekley]

It is by design that shotcut:detail uses absolute path; that is the detail. This is only used for annotation as a tool tip;

Hm, OK. But I think, if shotcut:detail is only a tooltip and is always necessarily the absolute path from the resource field, does it have to be serialized in the file? I mean, you could just infer the absolute path at runtime and not save it in the file, right?
And, if that's not the case (if the tooltip can't be inferred) could you at least not regenerate it with every change? Please? Currently it's impossible to manually remove that info from the file, because it's generated again.

it is not used to locate files.

The reason I'm asking for this is not only portability across computers, but especially privacy (suppose a user might have projects under C:\Users\John Doe\Documents). I believe that, in general, leaking absolute paths of relative files (without the user realizing) is not a good thing to do in any file format, because it can be a privacy issue for some.
I understand not everyone cares about this and a video editor might not tipically be used collaboratively (in a version control or Google Drive, etc) but it's nice to allow this possibility, right?

[geekley]

I do not plan to use a dialog; rather, there will be options to copy or move in the New Project view, and then it just happens always for that project.

So the users can use the option to have only relative paths, and by using the "link" option they can still reference both internal and external resources, right? Sounds great!
It could be something like: Import Mode = Copy Files | Move Files | Link to External Path.
I'd like to suggest placing these copied/moved files in a subfolder, for example, res.
I would also suggest a small text (in New Project view) explaining that the link option makes the project not portable across computers, or something along these lines.

And if it's a project setting, I assume we would also be able to change it for an existing project as well, somewhere in the UI, like in Settings menu, and changing that would only affect new things you import, not what you already have. Is that correct?

[geekley]

Relative paths to media assets on the in the XML should be working for files in the project folder. Provide an example MLT XML file with the problem or the steps to circumvent or break this.

Yesterday I had tried many things to reproduce it, and in the end I think this is what happened to me:

• Create project at folder1/myproj;
  myproj folder will be created
• Put a resource outside the project folder, for example, folder1/myres.png
• Open it in Source tab and drag to add it to Timeline and Playlist
• Save and close Shotcut;
  myproj.mlt will be created with absolute path references
• Suppose the user decided only now to put the resources inside the project folder:
  Move the file to inside the project folder, for example folder1/myproj/images/myres.png
• Reopen the project, it will warn about the missing references
• Double-click and fix the reference to point to the now relative path;
  folder1/myproj/myproj - Repaired.mlt will be created;
  note that the resources are being referenced with an absolute path, despite being relative (probably because the repaired .mlt name is different from the name of the parent folder)
• Close Shotcut;
  myproj.mlt still contains the invalid paths
• Suppose the user only now noticed the repaired file is different from the original (because there is no popup warning saying that) and decided to delete the original .mlt and rename the repaired file to replace the original; now the original file has the absolute references too
• If the user moves the project to folder2/myproj, now the references are invalid

Maybe this is not a big problem because, if I had changed anything normally through the UI (before moving to folder2) or used "Save As...", the paths would have been fixed automatically. Also, after you move to folder2 you can still fix it again through the UI. And I think the solution of having this import mode (with copy/move/link options) will already make it unlikely to happen now.

However, I still think it's worth also fixing this by changing the repair behavior to something less confusing for the user: instead of saving a myproj - Repaired.mlt file and switching to it, you could just repair the original project without saving it, mark the file as modified and let the user manually save it after they verify everything is correct. So they can still continue to work normally without having to use "Save As..." or switch to the original file.
If you want to keep the unrepaired file as backup, you can also save that one with a different name like myproj.old.mlt for example, while still keeping the correct file open in the editor.

[geekley]

Oh, I might have been something else too! I did the previous steps (but I used save as... to fix the file, so have a normal file).

So I opened the file, dragged the myres.png from playlist to timeline, in front of the existing clip, so leaving a blank space between the previous clip and new clip, and saved.

Now the project file is using absolute paths in the resource property of all producers (and mixing \ and / slashes), and even has an absolute path for the "black" resource <producer id="black"> like this: absolute\path\folder1\myproj/black.

[ddennedy]

Yes, I had been suspecting the mixed usage of slashes is causing a problem. The code that ensures a path is relative is only doing a comparison at the beginning of strings to trim them and is sensitive to slash direction. The software for file operations does not care about the direction of slashes, but I know the forward slashes confuses some windows users. I will try a change to ensure all file paths on Windows use backslashes. That should address this issue (the main one raised) as well as the paths displayed to users.

[ddennedy]

I will try a change to ensure all file paths on Windows use backslashes.

This has changed to always use forward slashes in the XML and trying to always use backslashes in the UI. The XML needs forward slashes to be compatible with other OSes in the case of using relative paths for portable projects.

[geekley]

I made a quick test and it seems everything is working as expected (XML has only forward slashes, relative paths and no absolute path unless its an external file). Thanks for the fix! 🎉

Do you still plan to implement this too?

users import a file located outside the project [...] there will be options to copy or move in the New Project view, and then it just happens always for that project

What about this one? I think it's worth doing at least this one. Should I open a separate issue?

changing the repair behavior to something less confusing for the user: instead of saving a myproj - Repaired.mlt file and switching to it, you could just repair the original project without saving it, mark the file as modified and let the user manually save it after they verify everything is correct

[ddennedy]

Do you still plan to implement this too?

Eventually, but that is an enhancement separate from this, and we do not track enhancements or feature requests outside of a pull request.

repair the original project without saving it, mark the file as modified

I do not agree. We have a lot of primitive users who will complain that they saved their recovered project, and now things are all messed up with no way to recover or try again. The current approach adds a crude form of versioning/backup.

[geekley]

The current approach adds a crude form of versioning/backup

I 100% agree with keeping a backup. Like I said before:

If you want to keep the unrepaired file as backup, you can also save that one with a different name like myproj.old.mlt for example, while still keeping the correct file open in the editor.

What I meant is: the version with a different name is supposed to be the backup (the old broken version), not the repaired version that you intend to keep working on (which is presumably correct). That would be less confusing for the user.

The reason for this is: messing it up and needing to restore a backup is the exception. The norm is either:

• you use the dialog to repair the file, check if it's working, save and keep working on it; or
• you manually put the missing files on their original paths externally, then try to reopen it, see if it works now, then keep working on it

The only problem with the current approach is that it's counter-intuitive: after repairing the file, the user still have to realize that the right file name is different now, then close the editor and manually rename the project back (after deciding to delete or rename the backup as well, since it has the original name) and then reopen the correct file.

[geekley]

Oh, I see now that I might have phrased "it" wrong. When I said

repair the original project without saving it, mark the file as modified

I meant without saving the repairs automatically. Just copy the original as a backup and mark the repaired version (still open in the editor) as modified, so the user can decide to manually save it if its right.

[ddennedy]

I understood what it meant, and I agree it is a little less friendly as far as naming goes today. But the current approach is safer and already works. It makes no attempt to modify the existing project file whatsoever. You are proposing to rename it. What if that fails? It might be locked by some process - especially frequent on Windows. Then, there needs to be another remedy, and these two new operations need testing and are still likely to have a bug at some point as it's additonal code. I have better things to work on. Feel free to work on this change yourself; it would be appreciated. I might come back to it sometime later tho when it bothers me too much.

[ddennedy]

Just copy the original as a backup

OK, that is safe and then try to let the modified in-memory project overwrite the old file.

[arzpace]

I've downloaded the 19.04.30 version, but still have this issue. I created and edited a project in an external hard drive and saved the file. Then I brought this hard drive to another computer and tried opening the same project file, the missing files window showed up, all files had absolute path...

Version: Windows 7 64-bit; Shotcut 19.04.30

[geekley]

@arzpace Was Shotcut already in 19.04.30 when you saved the project in the old computer? If yes, then we might still have some issue. If not, that dialog will have to appear.

For anyone with this issue: if you have lots of paths to fix and don't want to do it manually in the dialog you have 2 options:
a. Open it in the old computer, then open and save it in 19.04.30 version, and then it will be portable.
b. Or, edit the MLT file in a text editor, replacing absolute paths with relative paths (you can easily do that with CTRL+H then replace prefix in all paths with nothing), and also replace \ with /. The shotcut:detail property tags can be simply deleted, I believe.

Note that if the project has references to files outside the project folder, the path will be absolute regardless. That can be a problem if files are in an external drive - the path might change from D: to E:, for example, or be completely different on Linux, etc. That's why I would recommend to always move or copy the file to inside the project folder.

[arzpace]

@geekley Thank you for the explanation! I think it could be a good idea to add a function which can let the user to re-select only the first path in the dialog, and all other missing file with the same path will be redirected to the new path automatically. I think Blender can do something like this when re-link missing files. The portability is important for educational purpose, because I always need to share files with students.

[geekley]

I think it could be a good idea to add a function which can let the user to re-select only the first path in the dialog

Well, now that local references are saved properly with relative paths, implementing the "import external files copy/move into project folder" feature would already mitigate or prevent this problem from even happening in most cases. If that was done, I think this becomes lower priority (because it would become an exceptional case).

However, if this change in the dialog is implemented anyways (just for the sake of improving the feature), here's what I think would be the best way to do it. You could group all paths that are under a common folder (possibly in a tree structure) and allow to re-locate the whole folder at once, while still allowing to override the path for specific files.

For example:

• D:/Projects/MyProj/ -> C:/Projects/Shotcut/MyProj/Video1/res/
  • ./part1.mkv -> not set, so automatically under res folder
  • ./part2.mkv -> not set, so automatically under res folder
  • [...]
  • ./outro.mkv -> C:/Projects/Shotcut/MyProj/_common/outro.mkv (override)
• D:/Images/logo.png -> C:/Projects/Shotcut/MyProj/_common/logo.png

A tree structure would allow relocating files efficiently regardless of the folder structure (or lack thereof) the user had, although it might be an overkill for the problem (since it's very likely that a path change is the same in all files). So you could instead just group the 1 common dir among all files.

But again, this might not be necessary anyways, because, when people start using the 19.04.30 version and start importing files with relative paths (placing them somewhere under the project folder) then missing paths will become a lot less likely to happen.

[geekley]

The portability is important for educational purpose

I agree, and I'd like to add that I also think the program being explicit and transparent in everything it does is important for this - that's why I really like that the authors decided to use XML for the file format.