[Feature Request]: Import from local file

[valankar]

Feature Description

I have some large mbox files I'd like to import which are on the same machine running the docker container. I'd like to avoid needing to upload this via the web. Is there a way to import via local docker volume, or perhaps a feature can be added? Thanks.

Proposed Solution

Allow to browse local volumes for mbox file to import.

[hammaschlach]

You already can import mbox and eml.

[valankar]

I mean without needing to upload via the web interface. For a large mbox, I have them on the docker server itself that I'd like to import. Is there some command to do that?

[s1t5]

Currently the import can only be done by uploading the file, i've added this feature request to the roadmap

[brunohpg]

I have some EML file backups to import. One here is over 200 GB.

I think importing from the local file system is the best way. Okay, that was obvious in this discussion.

But, my data is in tgz. I tried importing other tgz files and saw that there is no support. I saw that dotnet has TarReader, so it's possible to import in stream, without needing to decompress all the files. However, I don't know if that would work because of the metadata files.

Anyway, it would be interesting to evaluate tgz. If that's not possible, at least an option to import a folder into the file system instead of just zip files. If I define a path and extract the tgz locally, I wouldn't need to create a zip file afterwards.

[s1t5]

Short update: The feature is currently in implementation!

[s1t5]

Hi everyone!
I've created an initial version for local file import and published it under the dev tag of the container.
Feel free to test it with a larger file, as I only have access to relatively small archives at the moment.

The documentation for this feature can be found here: https://github.com/s1t5/mail-archiver/blob/main/doc/CLI-Local-Import.md

[brunohpg]

I tested it. Since my file was an emls tgz file, I had to decompress and re-compress it into a zip file. This took about 10 hours (I guess my machine isn't very good).

The decompressed files were 370GB and the zipped files were 228GB.

Afterwards, I tried to import it, and it seems to have gone smoothly, with a smooth load, without excessive memory or CPU usage.

Here are the results:

=== Import Results ===
Status: Completed
Total Emails: 562560
Imported Successfully: 468096
Failed: 3
Skipped (duplicates): 94461
Duration: 1.16:42:13.3180925

Postgresql folder now has 106GB.