Export with Subfolders (recursive) for EML, HTML, PDF and Plaintext - Testing and Feedback

[cleidigh]

@kiki-ja
@tidebre67
@DenB10
@librelegal
@warrendodge
@pe4000
@zwergneun
@Thorsten-Nutzer
@projecttocs
@grzegorzsurowiec
@Yarny0
@Mr-Update

All
I have tagged you as a power use, someone I have done a fix or tweak,
translators or all the above!

Adding subfolder ( recursive ) export is a way overdue addition.
I had to patch it in to code that all has to be rewritten
for the next esr. It's pretty important it gets as many
eyeballs on it as possible. Thanks for any testing.
If you are not interested, no problem, just unsubscribe.

Translators:
I have not finished the English so I will post when that is ready.

Notes:

I made a change to the output structure for both single
and recursive exports, this is for HTML, PDF and plaintext :

Subfolder export of Inbox:

Inbox_20240225-1330
Inbox
index.html
messages
msg1.html
msg2.html
Attachments
Subfolder1
index.html
messages
msg1.html

For EML I did it differently so that the structure matches
the required directory structure for importing EMLs:

Inbox_20240225-1330
Inbox
index.html
msg1.eml
msg2.eml
Subfolder1
index.html
msg1.eml

You can import at the top container level and you get the folders created even if you import
at the account level.

For PDF with attachments we will have to wait until I
do my first rewrite for the next esr using more webextension apis.

Any comments or suggestions welcome.

v14.0.2-b10
https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b10-tb.xpi

Cheers,
Christopher

[DenB10]

@cleidigh

I've noticed a problem when exporting sub-folders in HTML (or PDf or plain text): the "messages" folder is not created in the sub-folder.

Inbox_20240225-1330
Inbox
index.html
messages
msg1.html
msg2.html
Attachments
Subfolder1
index.html
messages
msg1.html

In this way, messages contained in the subfolder are inaccessible by clicking them in the index (wrong path).

Cheers,

Denis

[cleidigh]

@DenB10
Is there any chance that the option :

• create a new directory with an index for messages
• on the Export directories tab, is turned off?
• Do you get a top container directory?
• Can you capture the console?
  Surprised as I did a million exports...
  Christopher

[DenB10]

@cleidigh
I've uninstalled then reinstalled IETNG, cleared the cache, the options are at their default values (the option you mention is enabled).
I sent myself 3 messages, I put one of them in a sub-folder. I still get the same result when exporting to HTML:

Courrier entrant_date
Courrier entrant
messages
message1.html
message2.html
sous-dossier
message3.html
index.html
index.html

message3.html is not in a folder "messages".

I'm attaching a copy of the console:
console-export-2024-2-27_21-28-0.txt

[cleidigh]

@DenB10
what about
messages
that's not a folder?

[DenB10]

@cleidigh
"messages" is an explorer folder containing html messages, isn't it???

[cleidigh]

I assume it has the top two messages?
So I think I understand it's the second level that lacks the messages directory?
C

[DenB10]

That's right.

[cleidigh]

@DenB10
Thanks. It's hard to do the trees because the indents go away!
@cleidigh

[DenB10]

Thanks. It's hard to do the trees because the indents go away!

In fact, I haven't figured out how to make indents. Sorry.
Maybe this file is more comprehensible:
FileList.txt

[cleidigh]

@DenB10
You can use the windows tree command
tree /f
Then paste between three backticks, that gives a nice tree.
Christopher

[DenB10]

@cleidigh
Like this?

D:\Users\DenB1\Downloads\Courrier entrant_20240227-2127>tree /f
D:.

+---Courrier entrant
    ¦   index.html
    ¦
    +---messages
    ¦       20240227-message1-19959.html
    ¦       20240227-message2-19963.html
    ¦
    +---sous-dossier
            20240227-message3-19971.html
            index.html

Thanks for teaching me something 😉

[cleidigh]

@DenB10
Great little tool.
Should use graphics characters, not sure why yours did not.
@cleidigh

[pantropia]

Helo! I would very much like to try this beta version but I can't see how to download it. There's a download button but it doesn't seem to do anything...

[cleidigh]

@pantropia
Excellent!
I literally just posted b11 with a bug fix and full localization.
You just click on the link below and then install from the addon page by file.
https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b11-tb.xpi

What formats do you use?
@DenB10
This fixes the messages directory bug.
@cleidigh

[pantropia]

Ah. Apparently I should have been clicking the Raw button not the Download one. Obvious!

[cleidigh]

@pantropia
Were you able to install and run?
@cleidigh

[DenB10]

@cleidigh

This fixes the messages directory bug.

Yes, it works!
You even translated the menu items for me, which is wonderful. Life is good.
Thanks for everything.

[cleidigh]

@DenB10
Excellent!
Yup, Google did the hard work %-)
@cleidigh

[pantropia]

@pantropia Were you able to install and run?

Yes it looks like it worked but I'll need to do some faffing about to determine whether it's done it all properly.
I'm basically wanting to export everything to .eml files both because I don't like backups I can't see the contents of without restoring them and because I know there's a load of duplicated stuff in various accounts due to a mishap some time ago which I'd like to sort out.
I tried EMClient for a while (didn't like how it does rules) but that was able to export all mail from all accounts/folders to a given location in a single operation. I couldn't see any way to export whole accounts to .eml with this, never mind doing all accounts at once, which is a shame, but recursive subfolders is still a considerable improvement over what I would have had to do without it - one old account has folders many layers deep for a selection of mailing lists which had been subdivided into subfolders for year and then month.
To know if it worked properly, I'll need to compare the export to what came out of EMClient, but because the structures are different I'll need to either spend some time amending one of the structures to match the other before using a sync tool to compare them or use a duplicate finder (the one I use doesn't do multiple simultaneous sessions and is currently trying to deal with a 3TB folder off my old NAS).
Before I start messing around I'll want to look into how the import from .eml files works, as I don't want to end up creating a perfect deduplicated and properly organised copy of my mail which I then can't get back into Thunderbird!

[pe4000]

Hello Christopher,
the function: "Export messages in → plain text format → messages and attachments"
no longer works on my computer in version v14.0.2-b10.
Greetings, Peter

[DenB10]

@cleidigh
@pe4000

I agree with Peter
Indeed, I've just noticed that all the functions for exporting selected messages in both plain text and HTML have been broken since beta 10, i.e. since the successful addition of the recursive export.
More work for Christopher, sorry! 😢

Edit: On further investigation, copying an entire message to the clipboard no longer works either.
Something's wrong with exportTools.js

@DenB10

[cleidigh]

@DenB10
@pe4000
First everyone should be on b11

https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b11-tb.xpi

I am not seeing issues with html or plaintext options. Perhaps more details and a console grab?
I did reproduce the copyToClipboard issue. No idea how that broke...
@cleidigh

[DenB10]

@cleidigh
Sorry but I'm actually using beta 11 and I'm seeing these issues.
Console copy:

menu update  
Object { menuIds: (30) […], contexts: (2) […], viewType: undefined, editable: false, pageUrl: undefined, selectedMessages: {…}, displayedFolder: {…} }
wextMenus.js:1140:11
check wextMenus.js:1295:11
tb.ui.interaction.message_display - The key length must be limited to 72 characters.
Object { menuItemId: "msgCtxMenu_Exp_HTMLFormatSaveAttsCreateIndex_Id", parentMenuItemId: "msgCtxMenu_Exp_HTMLFormat_Id", viewType: undefined, editable: false, pageUrl: undefined, selectedMessages: {…}, displayedFolder: {…}, modifiers: [], button: 0 }
 
Object { id: 1, index: 0, windowId: 3, highlighted: true, active: true, status: "complete", width: 866, height: 357, cookieStoreId: "firefox-default", url: "imap://denb10@imap.free.fr:993/fetch%3EUID%3E/INBOX%3E20036", … }
wextMenus.js:846:11
expMenuDispacher: winId 3 
Object { command: "WXMCMD_HTML_Format", params: {…} }
expMenuDispatcher.js:48:10
expMenuDispacher focused:  true expMenuDispatcher.js:49:10
ChromeWindow chrome://messenger/content/messenger.xhtml
expMenuDispatcher.js:50:10
TypeError: msgFolder is null exportTools.js:1426:2

@DenB10

[DenB10]

Maybe a screenshot is more readable.

[DenB10]

@cleidigh

Another issue I just realized:
With b9, I could import eml via the context menu of an imap folder, not with b11!!!! The menu item has disappeared.

[cleidigh]

@DenB10
line 1426 is a debug statement, that causes the copyToClipboard to fail, I found that.
It makes no sense for msgFolder to be undefined for any export.
I will have to add debug I guess.
@cleidigh

[cleidigh]

@DenB10
So your output is missing a lot of debug, it doesn't seem to follow the normal path.
Did you reboot?
@cleidigh

[cleidigh]

@DenB10
Ok I realized you were doing a selection export.
I have only been doing folder exports for now.
@cleidigh

[DenB10]

@cleidigh

Sorry for the late reply (I was asleep).
I didn't make any selection in the console entries. I just deactivated a very verbose extension and emptied the contents of the console display, performed the HTML (or plain text) message export operation from the email context menu and copied the console contents.
Let me know if I need to do anything special (develop an entry, ...).

However, I've just noticed something strange that solves all the problems: if I inactivate this famous line 1426, exporting becomes possible, as well as copying to the clipboard. If that helps...
Edit: No, not all: still impossible to import .eml files into an imap folder (no menu item).

@DenB10

[cleidigh]

@DenB10
Yes, the debug line 1426 for subfolders has a msgFolder which is not defined for the selected messages or copyToClipboard so I inadvertently broke those. Easy fix. Will post later.
@cleidigh

[DenB10]

@cleidigh

Since beta 10, I can no longer import .eml files from the context menu of imap folders, as the appropriate menu items no longer appear.
It's a function that's very important to me.
I've found a workaround by inactivating lines 1304 and 1308 in the wextMenus.js file, but I don't know if it's right.

@DenB10

[cleidigh]

I was fixing another menu issue that disabled the eml option

…

On Sat, Mar 2, 2024 at 4:04 PM DenB10 ***@***.***> wrote: @cleidigh <https://github.com/cleidigh> Since beta 10, I can no longer import .eml files from the context menu of imap folders, as the appropriate menu items no longer appear. It's a function that's very important to me. I've found a workaround by inactivating lines 1304 and 1308 in the wextMenus.js file, but I don't know if it's right. @DenB10 <https://github.com/DenB10> — Reply to this email directly, view it on GitHub <#538 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGAZ7625QOEDGMRZRY4L3ATYWI5GZAVCNFSM6AAAAABDZFVZ42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZUHEYDSMBRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cleidigh]

@DenB10
Got everything fixed, will post tomorrow after testing.
@cleidigh

[DenB10]

@cleidigh
I did a quick test.
All functions seem to work as expected.
To be confirmed by other users?
@DenB10

[cleidigh]

@DenB10
@pe4000
B12 fixes:

• selected mssg export
• copyToClipboard
• eml imports to imap
• various locale fixes
• support for old mac CR terminations

https://github.com/thunderbird/import-export-tools-ng/blob/v14.0.2/xpi/beta/import-export-tools-ng-14.0.2-b12-tb.xpi
@cleidigh

[DenB10]

@cleidigh

Thank you 🥇

[cleidigh]

@DenB10
Merci mon ami, c'est tout bien.
No more additions so I think I have to do some good regression testing. You have seen how easy it is to break things. Just have to see if user will complete cz locale contribution. Thanks for all The help!
Christopher

[Yarny0]

Tested with

• Thunderbird 115.8.0, Linux on x86-64
• IET NG v14.0.2-b12

Some changes to IET default settings (likely irrelevant for this test report)

author.max_length: 32
autobackup.dir: "/tmp"
autobackup.save_mode: 0
clipboard.always_just_text: true
csv_separator: "|"
export.cut_subject: true
export.filename_date_custom_format: "%Y%m%d-%H%M"
export.filename_pattern: "%D-%a-%s"
export.filenames_toascii: true
export.format_warning: false
export.help.showOnInstallAndUpdate: false
export.import_warning: false
export.index_date_custom_format: "%Y-%m-%d"
export.mbox.use_mboxext: true
exportEML.dir: "/tmp"
exportEML.filename_format: 2
exportEML.use_dir: true
exportMBOX.dir: "/tmp"
exportMBOX.use_dir: true
exportMSG.dir: "/tmp"
exportMSG.use_dir: true
recipients.max_length: 32
subject.max_length: 48

with four accounts

• IMAP/maildir (emails also stored locally)
• POP3/maildir
• POP3/mbox
• IMAP/mbox (emails also stored locally)

each with an Inbox with four mails (one with attachemnt) and one subdirectory, again with four emails (one with attachment).

I tested some export function in the context menu of each account's Inbox. These export functions generated results that look reasonable:

• single mbox file
• single zipped mbox file
• mbox, structured with subfolders
• mbox, with flattened subfolders
• all messages / html / messages & index & attachments with subfolders

Note: I observed that the zip export does not work with IMAP if mails are only stored on the server. However, there is a warning about this, so I don't consider this as a bug.

I am using mainly the EML export functions; these export functions produce exactly what I expect them to produce:

• all messages / eml / messages & index
• all messages / eml / messages & index with subfolders

---

Suggestion for improvement: When exporting a directory structure as HTML, a directory's index.html could also list subdirectories and link to the corresponding index.html files, like it lists and links messages. E.g. in

Inbox
+ Subdir
  + index.html  <-- links to msg1 and msg2
  + Messages
    + msg1.html
    + msg2.html
+ index.html  <-- links to msg3 and msg4, could also link <a href="Subdir/index.html">Subdir</a>
+ Messages
  + msg3.html
  + msg4.html

---

IET is a very valuable tool for me. Thanks for keeping up the development!

[DenB10]

@Yarny0

First of all, I'm impressed by your rigorous testing and presentation - well done!

Secondly, I had also thought of your suggestion for improvement, but I wonder whether the added value provided is sufficient, bearing in mind that any improvement is likely to break other functions.

But if it's easy to implement, why not? It's up to Christopher to decide!

[cleidigh]

@Yarny0
I second @DenB10 comment, your detailed report is very helpful and things look good.
I like your suggestion. It is actually pretty difficult in the current code structure because each folder export is separate and the structure is not understood by any other. However, I have to rewrite all this old code. At that point I will make the index have more of a global view which would facilitate this.
@cleidigh