[Feature]: bulk import should not report duplicates as import errors

[kylegoetz]

Is your feature request related to a problem?

I imported over 100K messages via an EML ZIP file. The results reported something like 2/3 of them as errors.

I investigated further

1. Exported the Mail Archiver email table (specifically the RFC822 message IDs)
2. Extracted the same header from each EML file

Iterated over #2, displaying the ones that didn't exist in #1. This produced 62 results, not the over 100K reported by Mail Archiver's batch EML ZIP import.

I then spot-checked EML files to see if they were in the archive. Every single one I checked at random was in Mail Archiver and discoverable with on the search page.

Finally, when I grepped Mail Archiver's output during the job, I saw a lot of "did not import because it already exists" messages. IIRC these were "warn"-level entries.

I can only conclude from this that Mail Archiver includes in its failure tally any email that already existed in its store.

1. Is this the case?
2. If so, I do not think this is the right behavior. I imagine other users will be left wondering why so many failures, afraid to delete their EML files for fear they will be deleting un-imported emails.

I do not consider "already existed" to be an import failure. It's a non-event.

Describe the solution you'd like

I propose that the batch import results list three data points:

1. imported
2. failed
3. already existed

Additional context

The EML files were produced from another archive program that I used to run but doesn't work with Gmail anymore. I last used this program in 2019, and finally got around to setting up a new archive (Mail Archiver) and wanted to merge in these old emails that, once backed up, I deleted (or, rather, thought I deleted) from Gmail.

[farfromexact]

Your investigation strongly suggests that duplicates are currently being counted in the failure total, even though the messages are already present and discoverable. That is a reporting problem, not an import failure.

I would expose three mutually exclusive outcomes for each EML file:

imported       a new message was stored
already_exists the message identity was found and no write was needed
failed         parsing, validation, storage, or another real error occurred

The summary should report all three counts, and a duplicate-only batch should exit successfully. Duplicates can still be logged at info level, but they should not be mixed into the error stream or make the UI look as if two thirds of the archive was lost.

For a large import, the result should also export the actual failed filenames and a reason per file. Then the safe deletion rule becomes:

imported + already_exists + failed = total input

Only the failed set needs another attempt. The identity check should remain idempotent, using RFC822 Message-ID when present and a stable content hash or equivalent fallback when it is absent. That preserves the useful behavior of re-running an interrupted import without creating duplicates.

The manual cross-check you did is exactly the kind of evidence that should become an automated regression test: a batch containing existing messages must produce a nonzero already_exists count and a zero failed count.