Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Export to Ardour #666

Closed
ratherlargerobot opened this issue May 18, 2019 · 14 comments
Closed

Feature Request: Export to Ardour #666

ratherlargerobot opened this issue May 18, 2019 · 14 comments
Labels

Comments

@ratherlargerobot
Copy link
Contributor

@ratherlargerobot ratherlargerobot commented May 18, 2019

This is a feature request, but I'm also volunteering to do a lot of the work to implement it. The basic idea is to add functionality to Flowblade that lets it export all of the necessary audio into a new Ardour project.

I'm including a lot of extra details here that you probably already know, as background information about the motiviation for this feature.

On a typical narrative film or TV project, the person doing the editing is usually not the same person that is doing the sound editing. I have personally worked on a few small indie projects that used a workflow modeled after the one used on larger budget projects. In a nutshell, that workflow (as it relates to NLE/DAW interaction) is:

  1. Film editor edits the project (on Avid, Final Cut Pro 7, Premiere, etc.), until it is stable enough for picture lock (e.g. no more edits that change the timeline or sync relationships between picture and sound).

  2. Film editor exports all of the audio from the project timeline in the OMF or AAF format, which has enough information to import the state of all the audio clips from the NLE into ProTools (or Logic or Nuendo or whatever), so that they appear as individual clips in the DAW. The clips are positioned in sync precisely where they were in the NLE, and also include "handles", which allow the sound editor to extend the ends of the clips past where they were in the NLE. They may or may not have any volume or fade information from the NLE, but the important part is getting the right clips in the right places with the proper in/out points in the DAW.

  3. Film editor also turns over the complete set of raw audio dailies to the sound editor, along with an exported work print video with no sound and a timecode burn. (The workprint and exported sound have a SMPTE leader and tail with 2 pop, so that the sound editor can confirm that the sync relationship has been maintained between systems).

  4. Sound editor imports the OMF or AAF into their DAW to create a new project.

  5. Sound editor opens the workprint video in their DAW and verifies that the sync relationship has been maintained.

  6. Sound editor (or sound department) performs dialog editing, adds music, sound effects, etc.

  7. Sound editor exports a final mix, with the 2 pop and everything still intact.

  8. Film editor receives the final mix from the sound department, and combines the final mix along with the final color graded edit with all VFX shots and credits and whatnot, to export the final release print.

My initial thought was that Flowblade should export OMF and/or AAF files, since that's already an industry standard, and would be compatible with a lot of DAWs and other NLEs. However, these formats are undocumented, and seem to be based on some sort of Microsoft structured storage format. And even then, they're haphazardly implemented in the commercial programs. Getting a given NLE to export an OMF or AAF that a given DAW can import is fraught with trial and error, and frequently requires multiple tries (involving communication between two different departments) to get it right.

Another thought was to define a generic open source format, and then see if we could get various other projects to use it. With luck, perhaps it would be adopted by the commercial systems later on. However, that seems fairly ambitious, and requires at least two projects to implement it before you'd see any results.

Getting Flowblade to export structured data to Ardour seems like it would at least be useful for any indie projects that wanted to use open source software though.

Here is an outline of one possible way that something like this might work. I'd love to hear your feedback, both on the general idea, as well as any hints you could offer about the subsystems involved in the code.

It seems like the File -> Export submenu could be extended to include an Ardour target. If there were a lot of different exporters eventually I can imagine a possible additional dialog to choose the export type, but this seems like a simpler way to get started initially.

Once File -> Export -> Ardour is clicked, it would open an Ardour Export Dialog, with a variety of settings. Ardour projects are directories, so you would have to choose a target directory, and then the name for the export. So, if you chose your home directory as the target, and the export was named "foo", then eventually Flowblade would export a directory called "foo" with all of the Ardour project files and subdirectories inside of it.

The Ardour Export Dialog would also need to include the audio sample rate. I'm not sure if Flowblade has a project-level sample rate, if it does, I couldn't find it.

Still in the Ardour Export Dialog, there would be an Export button. When the user clicks on this button, it would start the export.

The export would probably have to transcode all (or at least most) of the audio in the project. WAV format seems like a good choice. AIFF could also be an option, but just sticking with WAV as the only choice seems totally reasonable too. One less choice the user has to make, for something that doesn't make any significant difference. The Ardour default for recording files is WAV anyway.

Other NLE OMF/AAF export tools have the option of setting the length of the "handles", or the extra time on either side of each audio clip that is included in the export. For this first version, I propose not including this feature and just including the entire clip. Audio files aren't all that big, and in a workflow like this it's often helpful to have the extra sound as a potential source of extra room tone and such. Sound editors like long handles.

So then, the export would transcode all of the audio into new WAV files inside of the Ardour project directory, and create the XML files and whatnot necessary to create a fully-formed Ardour project.

Once this is exported, the editor would deliver the exported project to the sound editor (if they are different people), by putting it on a removable hard drive, tarring it up and sending it over the internet, etc.

Then, the sound editor could open this project in Ardour, along with an exported workprint, and start editing sound.

In terms of implementation, it seems like the other File -> Export stuff is a good place to look for inspiration. I haven't dug in too deeply yet, but it looks like Flowblade is using MLT XML files for these exports under the hood. So maybe an MLT XML file could be created to transcode the audio into WAV files? And maybe something like the MLT XML temp file that was used for, say, the EDL export, could be interpreted by the Ardour export code to figure out where to create the corresponding clips in the Ardour project. I'm sure I can figure something out, but any advice would be most helpful.

Your thoughts?

@ratherlargerobot

This comment has been minimized.

Copy link
Contributor Author

@ratherlargerobot ratherlargerobot commented May 19, 2019

After thinking about this some more, I think I'm going to start by creating a smaller external script first, as a prototype. It will read an MLT XML file (of the sort exported by Flowblade via File -> Export -> MLT XML). It will use ffmpeg to do the transcoding (if necessary). It will create a new Ardour project.

I'll post here when I have something to show. It might take a few weeks or so of hacking on it in my spare time.

@dvdlvr

This comment has been minimized.

Copy link
Contributor

@dvdlvr dvdlvr commented May 19, 2019

A while ago I've created a Renderfarm script that reads and manipulates the MLT XML file.
It may be easier to use this to get you on your way.
I've included the Renderfarm script as an attachment. Also two necessary config files are attached.
The script allows you to set an audio channel - provided the source audio channel is recognized by ffmpeg. Current setting is 'eng' in the config file.
I hope this helps.
RenderFarm.gz

@jliljebl

This comment has been minimized.

Copy link
Owner

@jliljebl jliljebl commented May 20, 2019

@ratherlargerobot thanks for the long description of the process and desired functionality.

I've done commercial editing with workflow including sound guy too, most of was just proving sound editor with a digital file with 2-4 track audio and audio zig-zagged into multiple tracks and some handles added so that sound guy could do fades between clips themselves to remove any pops in audio caused by straight cuts.

They had ProTools then display the video synchronized with audio as they worked on the mix. Then I got mix back and made master.They just ripped audio from the video and cut it in Protools if needed. These were mostly 15-30s commercials.

Here are my points:

However, these formats are undocumented, and seem to be based on some sort of Microsoft structured storage

I found a 100 page spec on OMF somewhere and decided never to touch it, it seemed a very bad format based on some C++ memory dumps if I remember correctly.

Another thought was to define a generic open source format...

I don't think this is the approach, these are very hard to get going.

Getting Flowblade to export structured data to Ardour seems like it would at least be useful

Yes, getting structured data in editor is clearly the best approach and I do believe this to be useful, just working on rendered audio will be an inferior workflow to this. I also believe Ardour is the best target for this.

It seems like the File -> Export submenu could be extended to include an Ardour target

Yes, this where this would go.

The Ardour Export Dialog would also need to include the audio sample rate. I'm not sure if Flowblade has a project-level sample rate, if it does, I couldn't find it.

In MLT (and therefore Flowblade) all project paramaters are these: https://github.com/jliljebl/flowblade/blob/master/flowblade-trunk/Flowblade/res/profiles/atsc_1080i_50

So no audio samplerate, needs to be set on export.

So, if you chose your home directory as the target,

I think home directory should be disallowed as the target

Other NLE OMF/AAF export tools have the option of setting the length of the "handles", or the extra time on either side of each audio clip that is included in the export.

Yes this is so that clicks and pops caused by straight cuts can be removed. But if all media is provided sound editor can drag clips longer in Ardour, right?.

For this first version, I propose not including this feature and just including the entire clip.

We will eventually need functionality that splices out audio into multiple tracks and adds handles, but this is orthogonal to Ardour export, we actually need this to help with mixes on timeline too.

but just sticking with WAV as the only choice seems totally reasonable too.

If think every DAW imports WAW so yes this is good enough.

I think I'm going to start by creating a smaller external script first, as a prototype. It will read an MLT XML file (of the sort exported by Flowblade via File -> Export -> MLT XML). It will use ffmpeg to do the transcoding (if necessary). It will create a new Ardour project.

This sounds all good. If we can get this I believe it will be a valuable feature that improves workflow for anyone trying to achieve more high end results with these tools.

2.2 is coming in couple of weeks and 2.4 will be Python3 port but if we can get this for 2.6 sometime in the autumn I'll happily add it.

@ratherlargerobot

This comment has been minimized.

Copy link
Contributor Author

@ratherlargerobot ratherlargerobot commented May 21, 2019

@dvdlvr Thanks for the render farm script, I'll check it out!

@jliljebl Thanks for your response, that all sounds very reasonable.

You said that the home directory should be disallowed as a target. What I really meant there was that if you picked your home directory as the directory to contain the new directory you would have to create, which would hold the ardour project. For example, if your export was named "foo", then you could select your home directory (or whatever), name the export "foo", and then ~/foo/ would be the newly-created directory which would hold the exported Ardour project. I completely agree that just splatting a bunch of loose files directly into your home directory would be a disaster.

As for the handles, you are correct. If you had a 10 minute clip in your bin in Flowblade, and one frame of it had audio included, you would end up with a 10 minute WAV file exported to Ardour, along with an Ardour timeline entry to place that one frame in the right place in Ardour. From there, you could drag the in and out point handles to extend the length of the clip, copy the clip and pull room tone from something five minutes before that, etc. My observation with OMF/AAF export in other tools is that if you pick, say, 30 seconds for handles, then you would get whatever was on the timeline, plus 30 seconds on either side of it, but no more than that. So in my previous "one frame" example, you would get the one frame, plus 30 seconds before that, plus 30 seconds after that, for a total of 60 seconds and one frame of included WAV file data. I'm simply suggesting that we skip all of that and include the entire clip as described above.

I'll hack on the prototype script over the next few weeks or so. I'll do it in Python 3, so it will be easier to adapt to Flowblade 2.6.

Thanks again!

@ratherlargerobot

This comment has been minimized.

Copy link
Contributor Author

@ratherlargerobot ratherlargerobot commented Aug 8, 2019

Unfortunately, I have been busy with other things. However, I still hope to get to this when time allows. I'll post here when there is some progress.

@jliljebl

This comment has been minimized.

Copy link
Owner

@jliljebl jliljebl commented Aug 9, 2019

Ok. Still haven't got 2.2 out and I'm currently doing some release and other testing, so there will little activity on repo, but I'll get this cycle wrapped in August.

@ratherlargerobot

This comment has been minimized.

Copy link
Contributor Author

@ratherlargerobot ratherlargerobot commented Oct 13, 2019

After not having a chance to even look at this for months, I spent the last few evenings hacking on this and made some progress!

I have created a command-line script called flowblade-ardour-export. It takes a Flowblade MLT XML file, Ardour project name, and project sample rate as input. It extracts all of necessary metadata from the MLT XML file, transcodes everything that contains audio, and creates an Ardour project file.

I'm currently going over and polishing it a little more, and trying it with more elaborate Flowblade projects. But it all seems to work.

The script is implemented in Python 3, and only has a runtime dependency on the ffmpeg command (to do the transcoding).

Should I submit a pull request to the Flowblade Github repo? If so, where would be the best place to put it within the project? Or, if there isn't a good place to put it in the Flowblade project, I could create a new repo under my own account and post the link.

I'd be interested to hear if it works for other people as well.

@jliljebl

This comment has been minimized.

Copy link
Owner

@jliljebl jliljebl commented Oct 14, 2019

Great to hear, excellent.

I think the /tools folder would good place to put it since it is a Python 3 script and communicates with an external tool.

Please, do submit a pull request and I'll pull, test it and think about the best GUI representation and code integration and comment here.

Looking forward to testing this.

@x42

This comment has been minimized.

Copy link

@x42 x42 commented Oct 14, 2019

Keep in mind that Ardour does not have a stable session-format. Although for the case at hand that may not be relevant.

The mid/long-term plan for Ardour is to support import (and export) from
https://github.com/PixarAnimationStudios/OpenTimelineIO

@ratherlargerobot

This comment has been minimized.

Copy link
Contributor Author

@ratherlargerobot ratherlargerobot commented Oct 14, 2019

Thanks @x42! I know the Ardour session format is a bit of a moving target. But newer versions of Ardour seem to be able to open projects created with previous versions of Ardour. It's not too much of a stretch to assume that if Ardour 5.12 can open the generated projects without any warnings or errors, that Ardour 6.0 has a good chance of opening them too, right?

I'll take look at Pixar OpenTimelineIO. I wonder if it makes more sense to have Flowblade export directly to OpenTimelineIO, or to add a Flowblade MLT XML input plug-in to OpenTimelineIO?

@jliljebl I'll submit a pull request later tonight, and put the script in the tools/ directory.

@jliljebl

This comment has been minimized.

Copy link
Owner

@jliljebl jliljebl commented Oct 14, 2019

@x42 @ratherlargerobot OpenTimelineIO seems to have a good change at becoming a widely used standard so we are also looking to support it medium term.

The workflow enabled by being able to export audio into Ardour is in my view quite important and once we get it going it will be worth it to put some time in to fix version churn.

ratherlargerobot added a commit to ratherlargerobot/flowblade that referenced this issue Oct 15, 2019
This is a standalone command-line script to convert a Flowblade MLT XML file
into a new Ardour project for post-production sound editing.

See jliljebl#666 for details

Usage: ./exportardour.py exported-flowblade-mlt.xml /path/to/new-ardour-dir

The sample rate (in hertz, e.g. 48000) can be added as an optional third
positional command-line argument.
@ratherlargerobot

This comment has been minimized.

Copy link
Contributor Author

@ratherlargerobot ratherlargerobot commented Oct 15, 2019

OK, I have submitted a pull request with the Ardour export script. I added it under tools/exportardour.py

The script accepts a Flowblade MLT XML file, the path to a new Ardour project directory (that does not yet exist), and an optional sample rate to use in the Ardour project (defaulting to 48000).

$ ./exportardour.py
exportardour.py
Usage: exportardour.py MLT_XML_FILE NEW_ARDOUR_PROJECT_DIR [SAMPLE_RATE_HZ]

To use this script:

  1. In Flowblade, perform an MLT XML export (File -> Export MLT XML)

  2. In Flowblade, without making any additional edits (at least none that affect any audio sync relationships), render the video in its current form as a work print.

  3. Run exportardour.py, reading in the XML file generated in step one, and specifying a new Ardour project directory (that must not exist yet, and will be created)

  4. exportardour.py will use ffmpeg to transcode all of the media it can find in the Flowblade MLT XML project, and create the new Ardour project.

  5. In Ardour, open up the new Ardour project.

  6. In Ardour, import the workprint video into the project (Session -> Open Video). Ardour will transcode the video, import it into the project, and show the video in synchronized playback using the embedded xjadeo program.

  7. In Ardour, verify that all of the expected audio clips came through, verify sync relationships, etc. If you use a SMPTE leader and tail, the 2 pop makes this easy.

  8. Do all your post production sound editing in Ardour, and then later export the finished mix and import it back into Flowblade, etc.

I have tested this on my computer, and kept tweaking it until it worked for me. There are some assumptions baked into the script that are somewhat likely to work most of the time, but could certainly not work if the underlying conditions change. When an assumption like this exists, there are usually comments in the source code highlighting them. For example, the script makes use of the ffmpeg map channel feature to extract the audio from video and audio files, and it currently assumes that the audio in a video file will be on stream 1, but will be in stream 0 in an audio file. There are probably files out there that violate this assumption though.

I tested this using an MLT XML script from Flowblade 2.0 from Github, and used the official Ardour project binary release (Ardour 5.12) to import the file. The starting point for the generated Ardour project file format is based on a minimal Ardour 5.12 project I created with a couple of channels. Then I read the XML file, and figured out how to generate something that looked similar. I also omitted some sections that seemed unnecessary (e.g. Ardour could just regenerate them).

The script is imperfect in many ways. But on my computer, I was able to export a couple of different Flowblade projects into new Ardour projects and get everything to line up correctly.

At the moment, the script is not integrated into Flowblade in any way whatsoever. However, it has a couple of top-level functions (to read the XML file and create the Ardour project) that could be called from a different module. There are probably numerous other ways that it could be enhanced and modified to better fit into the Flowblade UI. Providing feedback that could be incorporated into progress bars, more generalized handling of input media, etc. Never the less, I think it's a good start.

I'd love to hear about how it works for other people.

@ratherlargerobot

This comment has been minimized.

Copy link
Contributor Author

@ratherlargerobot ratherlargerobot commented Oct 24, 2019

Update: this was been incorporated into Flowblade on the master branch

@jliljebl

This comment has been minimized.

Copy link
Owner

@jliljebl jliljebl commented Oct 24, 2019

I'll close this as feature added and we'll consider further work on compound clips as enhancements that might take some time. Thanks to @ratherlargerobot for this feature addition of this nice feature.

@jliljebl jliljebl closed this Oct 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.