Feature request: Quick format for file containers

[Zythyr]

It would be great if users had the option to "Quick format" when creating an encrypted file container. The quick format option exist when creating an encrypted partition, but NOT when creating a file container.

I don't understand why the option to "Quick format" is not available when creating file containers. When creating a large file container (ex: 500GB), it takes on the order of hours (>24 hours) to format... Yes I understand "slow" format is more secure ,however, users should have option to decide...

This feature has been requested by others also in the past also (see links below).
#256
https://sourceforge.net/p/veracrypt/discussion/features/thread/7b996a4a/
https://sourceforge.net/p/veracrypt/discussion/features/thread/195a824c76/

[idrassi]

I have implemented this addition option in commit 7ef3d5e and I have uploaded the installer for version 1.24-Beta0 that includes it to https://sourceforge.net/projects/veracrypt/files/VeraCrypt%20Nightly%20Builds/.

Can you please test this version and report your feedback on the newly added check box in the creation wizard?

Thank you.

[Zythyr]

I have implemented this addition option in commit 7ef3d5e and I have uploaded the installer for version 1.24-Beta0 that includes it to https://sourceforge.net/projects/veracrypt/files/VeraCrypt%20Nightly%20Builds/.

Can you please test this version and report your feedback on the newly added check box in the creation wizard?

Thank you.

Firstly thank you very much for introducing this feature. I am very excited for it!

I just tested the VeraCrypt Portable 1.24-Beta0.exe. I am able to enable "Quick Format", however I think it is still doing the slow formatting. I tested on an external drive by creating 50GB container and "Quick Format" enabled. The VeraCrypt Volume Creation Wizard says "Done 100%", however it is still doing something in the background. In Windows Task Manager, I see the Disk usage at 98%...

[idrassi]

I have checked again and the quick format option works as expected (file container creation is much quicker since we don’t overwrite all bytes manually and you can verify this by the speed displayed by VeraCrypt).
What you are seeing is a Windows behavior that is outside of our control: Windows is flushing data to disk. Depending on the machine and on the disk type, flushing data can be slow and VeraCrypt can not influence this part.

[idrassi]

An additional clarification: quick format in VeraCrypt UI is not related to filesystem quick format. If the user chooses a filesystem for its volume (FAT, NTFS, ExFAT or ReFS), VeraCrypt ALWAYS uses the filesystem quick formatting regardless of the quick format option on the UI.
The quick format in the UI controls if VeraCrypt will fill all volume content with random data or not. Such random data filling can take a long time.

[Zythyr]

An additional clarification: quick format in VeraCrypt UI is not related to filesystem quick format. If the user chooses a filesystem for its volume (FAT, NTFS, ExFAT or ReFS), VeraCrypt ALWAYS uses the filesystem quick formatting regardless of the quick format option on the UI.
The quick format in the UI controls if VeraCrypt will fill all volume content with random data or not. Such random data filling can take a long time.

Thanks for the the follow up. I am still confused and I was hoping you can provide calcification.

So assume I have an external HDD with 500GB space and it is formatted as NTFS. I use VeraCrypt to create an encrypted "File container' located in the external drive with a size of 250GB and select "Quick format". When VeraCrypt creates the file container it actually does take up 250GB space even though these is no content inside it. I understand this part this part because we're creating a fixed size container.

Based on my testing, it seems the actual time it takes to create the file container is dependent on the container size and the write speed to the disk. For example if max write speed is 100 MB/s and the file container size is 250GB, then it takes around 2500s (~41min).

What I don't understand is why does it take so long to create the container? It is an empty container with no data in it, so it still doesn't make sense what is the actual data Windows is trying to write to the disk when it is "flushing data to disk". In is my understanding the empty file container should be just allocation a range of sectors that it will occupy. However, the actual bits are not changing because we didn't write data to it. And regarding the "fill with random data", it is my understanding that the idea of "Quick format" is that user doesn't care too much about security so it is safe to NOT to randomize every single bit, as opposed to "full format" where every single bit is randomized. It would make sense if we choose "full format", then every single random bit needs to be written to the file container, therefore explaining why the creation takes a long time. But for the case of "quick format', I am assuming we are deciding to avoid randomizing the bits.

[idrassi]

I will try to use simpler words to avoid any confusion.

If the user selects Quick Format, then VeraCrypt tells Windows to creates the file container with the choose size (e.g 250 GB) and nothing else! I this case, VeraCrypt doesn’t do anything with the file content.

If the user doesn’t not select Quick Format, then VeraCrypt tells Windows to create the file container AND VeraCrypt fills all the file content with random data.

Thus, as far as VeraCrypt is concerned, selecting Quick Format reduces the time of file container creation because there will be no content filling with random data.

It is easy to check that selecting Quick Format reduces the time of file container creation and this is a fact.

Any time that remains when Quick Format is selected is taken by Windows not VeraCrypt.

One remark: when VeraCrypt tells Windows to create a file with a given size (e.g 250 GB), VeraCrypt do this by calling a single function but Windows will need a lot of time depending on the filesystem users because Windows will have to reserve the physical space on the disk and deal with disk fragmentation. This explains why this simple operation can take a lot of time.

At the end, what is important is that the Quick Format option in VeraCrypt is doing what Is supposed to do: remove all unnecessary write operation on VeraCrypt side and using only the minimum steps needed to create the file container without any further security measures.

[Zythyr]

@idrassi Thanks for clarification. After further testing on my end, I came to the following understanding. Please correct me if I am wrong.

When the file container is created, there must be a fixed space allocated to that container (ex: 250GB). Therefore, even if the file container is empty, the OS detects the container to be allocating a fixed space (ex; 250GB). Therefore, when the container is created, even though Quick format is enabled, the OS has to physically allocate the fixed space by by writing that file to the hard drive. Thus the reason why it takes a long time to create the file container because the OS is creating the file on the hard drive.

[AncalagonTheBlack]

I wonder if this can be also achieved for the linux version of veracrypt

[ranvis]

This should be possible with SetFileValidData() after SetEndOfFile().

For expanding existing file containers, users can do it faster even on the current release, at their own risk.

1. Expand a container file size to say 100GiB (be sure not to truncate the file.)
   > fsutil file seteof test.hc 107374182400
2. Skip zeroing. Expanded area may contain contents of removed files of the disk.
   > sutil file setvaliddata test.hc 107374182400
3. Now use VeraCryptExpander to exand.
   It will complain of the actual container file size being larger than it should.
   Also you must expand some more Kbytes to actually expand the volume (for a backup header located at the end of the file. So by using fsutil, the old backup header become unreachable.)

[idrassi]

Thank you for sharing this idea.

It should be possible to implement this approach in VeraCrypt Expander directly in a way similar to "Quick Format" when creating volumes that will warn users about the privacy risks of this approach. I will look into it.

[idrassi]

I was able to implement @ranvis idea relatively easily by adding a "Quick Expand" option and it works well. See commit 74e14c0.
Of course, a warning is displayed when the user select this option since it has security implications if proper precautions are not taken.

Thus upcoming 1.24-Hotfix1 that should released in 1 or 2 days will include this feature.

[idrassi]

Feature implemented in 1.24-Hotfix1 released today.

[smallerxie]

I have checked again and the quick format option works as expected (file container creation is much quicker since we don’t overwrite all bytes manually and you can verify this by the speed displayed by VeraCrypt).
What you are seeing is a Windows behavior that is outside of our control: Windows is flushing data to disk. Depending on the machine and on the disk type, flushing data can be slow and VeraCrypt can not influence this part.

I have exactly the same problem with @Zythyr , and I don't think it's windows who should be blamed for its "flushing data" behavior as said @idrassi . I just tested to create a volume of 10GB with and without the option "quick format" on my external disk, the results are:

• Without Quick format (with random data written): Volume of 10GB created successfully, speed was about 40MB/s finished in 4-5 minutes.

• With "quick format" selected : the formatting was so "quick" in less then 1s but then blocked in "100% done", from the task manager I could see that the disk was continuously being written at the speed about 20-30MB/s. Waited for more than 5 minutes and shut down the computer to kill the process. (And once started, the process "VeraCrypt Format" is difficult to be killed from the task manager.)

I DO confirm that the function "quick format" has a bug in the version 1.24-hotfix1.

[smallerxie]

Finally I found this solution : I created a small volume (say, 1GB), and used the Volume Expander Tool to enlarge its size (to 2TB in my case) by choosing the option "quick expand" which works well (operation was finished in several seconds).
I believe that the funciton "quick expand" and "quick format" should have used the same technique.
So, please fix the bug in the function "quick format" to make it work as "quick expand", thanks.
P.S. this issue is to be reopened...

[Zythyr]

I have checked again and the quick format option works as expected (file container creation is much quicker since we don’t overwrite all bytes manually and you can verify this by the speed displayed by VeraCrypt).
What you are seeing is a Windows behavior that is outside of our control: Windows is flushing data to disk. Depending on the machine and on the disk type, flushing data can be slow and VeraCrypt can not influence this part.

I have exactly the same problem with @Zythyr , and I don't think it's windows who should be blamed for its "flushing data" behavior as said @idrassi . I just tested to create a volume of 10GB with and without the option "quick format" on my external disk, the results are:

* Without Quick format (with random data written): Volume of 10GB created successfully, speed was about 40MB/s finished in 4-5 minutes.

* With "quick format" selected : the formatting was so "quick" in less then 1s but then blocked in "100% done", from the task manager I could see that the disk was continuously being written at the speed about 20-30MB/s. Waited for more than 5 minutes and shut down the computer to kill the process. (And once started, the process "VeraCrypt Format" is difficult to be killed from the task manager.)

I DO confirm that the function "quick format" has a bug in the version 1.24-hotfix1.

@idrassi I do agree there may be some weird bug. When I do "Quick format", it hangs at 100%. When I click "Abort", nothing happens. The formatting does NOT stop. I see Task Manager continuing to write on the disk even though I clicked Aborted. When I do NOT enable"Quick format", clicking "Abort" works as expected.

[wrecclesham]

I can confirm that VeraCrypt 1.24 Update 4 has this bug. How can this take over a year to fix? The bug didn't exist in TrueCrypt and was introduced at some point in VeraCrypt's development!

This "flushing to disk" explanation makes no sense to me. WHAT data is being flushed to disk? And why doesn't Windows flush data to the disk with TrueCrypt but it does with VeraCrypt?

I had to reboot my PC because the "Abort" button did nothing. Even signing out of the Windows user account doesn't stop the disk activity. Very annoying bug.

I gave up trying to create a 1 TB encrypted container using quick format (HDD stuck on nearly 100% for hours,) and instead ended up creating an encrypted non-system partition, which worked INSTANTLY when using quick format and works at the expected speed (as dictated by disk write speed) when quick format was not checked.

@idrassi please do some testing with earlier versions and with TrueCrypt 7.1a before being so quick to assume this is Windows' fault. This is clearly a bug with the current version of VeraCrypt. If you don't believe us, try to abort a quick format and see for yourself that the Abort button does nothing while the disk continues to be written to at a high rate, as per Task Manager.

[ranvis]

For those who doesn't satisfied still, there are some good articles to understand what is happening under the hood.

Why does my single-byte write take forever? | The Old New Thing
On the various ways of creating large files in NTFS | The Old New Thing