Add Support for Cardbook

[GrantHorizons]

I'm using CardBook extension as the default TB Address book is very limited - but there seems to be no way to get SmartTemplate4 to use CardBook. A "Which Address book to use?" question would be useful! And which address book IN Cardbook too.
thanks!

[RealRaven2000]

Please note that I am using an XPCOM function for retrieving AB data - it is not clear to me where this comes from. SmartTemplate uses the Components.interfaces.nsIAbDirectory - here is the relevant code:

  {
    let abManager = Components.classes["@mozilla.org/abmanager;1"].getService(Components.interfaces.nsIAbManager);
    allAddressBooks = abManager.directories; 
  }
  while (allAddressBooks.hasMoreElements()) {
    let addressBook = allAddressBooks.getNext()
                                     .QueryInterface(Components.interfaces.nsIAbDirectory);
    if (addressBook instanceof Components.interfaces.nsIAbDirectory) { // or nsIAbItem or nsIAbCollection
      // alert ("Directory Name:" + addressBook.dirName);
      try {
        let card = addressBook.cardForEmailAddress(mail);
        if (card)
          return card;

...as far as I know the interface nsIAbDirectory is able to retrieve data from CardBook, but you have to make sure not having a matching email address in the standard AB. I assume that cards in the standard AB have precedence when using the XPCOM method. XPCOM is going to be deprecated in favor of web APIs (which aren't written yet) - I would highly recommend filing a bug to include support for CardBook in the planned APIs.

Thunderbird 78 is planned for release in June 2020, I am very doubtful all APIs will be replaced by that time, this means SmartTemplate4 will have to be rereleased as a "web experiment", which also requires a lot of rewriting. (realisticly between 4 and 8 man-weeks) - I am currently considering which route to take; probably a kickstarter to hire an external developer for the conversion work so that I can continue working on bugs, maintenance, new features and user support!

[RealRaven2000]

Going to pin this one for next implementation. Sorry for taking such a long time. It requires rewriting of pretty much all functions as "asynchronous".

[GrantHorizons]

It always seems so simple - but never is. Thanks for the update

[richanl]

I am also eagerly awaiting the integration of Cardbook fields into Smart template: #141

[RealRaven2000]

After such a long time, this still isn't implemented in 2023 - it's time this has to become a reality. I will need the help of Philippe (the author of Cardbook) in order to pull it off; I will ask him for some boilerplate code on our next Add-on Developer meeting (next Thursday).

[tkcantonedu]

I'd also love to see this integration. Both SmartTemplates and CardBook are powerful tools I use regularly.

[RealRaven2000]

Working on this right now, the very step is to make all calling functions async. which also means the main parser (in the experimental code) needs to become async. I have already done similar work in the (unfinished) web extension portion of the code (files starting with st-) so I am hopeful I can reuse some of that code.

The main module that pulled everything together for the "mx compatible" method (which is incomplete due to lack of UI cooperation) is st-parser.mjs.js.

[RealRaven2000]

List of functions that need to be async-ified - initial list as starting point:

getCardFromAB  - local function in mime.split, retrieves cards from the address book
SmartTemplate4.mimeDecoder.spli - this is doing all address field processing
Local functions [SmartTemplate4.regularize] 
simplify
classifyReservedWord
replaceReservedWords

SmartTemplate4.regularize()

classSmartTemplate.getProcessedText
extractSignature
classSmartTemplate.insertTemplate
insertFileEntryInComposer
loadIdentity
SmartTemplate4.init

smartTemplate_loadIdentity - this is a MonkeyPatch for the Thunderbird function LoadIdentity()

Problem with classifyReservedWord: this is passed as argument to a string.replace function - which by default expects a sync function. I believe I have already solved this very difficult problem in the mx code branch, just need to check how exactly it was done. I wrote a utility function replaceAsync in st-util.mjs.js - probably need to integrate this into my main SmartTemplate4.Util namespace.

this might be a major problem, (close to a showstopper) as Thunderbird surely calls this as a sync function. Need to check the Thunderbird source code, to see what it does "afterwards". Reminder - you can all an async function from a sync function, but you cannot "await" its result - the calling code will just continue relentlessly (even if LoadIdentity is still working or working later on).
Example for it being called by Thunderbird - during NotifyComposeBodyReady() it does some processing, then calls loadHTMLMsgPrefs() and AdjustFocus()
https://searchfox.org/comm-central/source/mail/components/compose/content/MsgComposeCommands.js#582

in replaceReservedWords():
modifyHeader

[RealRaven2000]

next to asyncify:

expand = function(str) { return str.replace(/%([\w-]+)%/gm, replaceReservedWords); };

also

getQuoteHeader
getSmartTemplate

Result so far:

Still getting a "promise" output in resulting mail, so I am probably missing at least one await somewhere. Also the missing from / to headers may be a problem of call order, not sure yet. Are these headers available at time of replacement?

[RealRaven2000]

First test version with async processing - still some problems with address headers but it's a start:

smartTemplate-fx-3.16pre31.zip

once async works reliably I can start tests with Cardbook's new interfaces.

[RealRaven2000]

Asyncified code some more, see below. THis is before adding any of Phillipe's functions (CardBook).

smartTemplate-fx-3.16pre36.zip

Will probably commit this changeset to ESR91 - I will test this version on my production profile for a day or two before doing the actual Cardbook notifications. Might have to add more async;ified code before that.

[RealRaven2000]

Structure of a card returned by cardbook (below). Spelling & names obviously different from normal AB format. I am missing displayName. Need to check how to translate to the legacy format or vCard (unless this is in vCard format already)

note the legacy card structure had a property "vCardProperties"

adr: Array []
agent: ""
anniversary: ""
bday: ""
birthplace: ""
cacheuri: undefined
cardurl: ""
categories: Array []
cbid: "143b253f-d955-4326-b0dc-ab161e558074::28ff0dbc-1803-44a6-86be-b980ae41fa76"
class1: ""
created: false
deathdate: ""
deathplace: ""
deleted: false
dirPrefId: "143b253f-d955-4326-b0dc-ab161e558074"
email: Array [ (4) […] ]
emails: Array [ "thunderbirddaily@gmail.com" ]
etag: ""
firstname: ""
fn: "thunderbirddaily"
gender: ""
geo: ""
impp: Array []
isAList: false
key: ""
kind: ""
lastname: ""
logo: Object { types: [], value: "", localURI: "", … }
mailer: ""
member: Array []
nickname: ""
note: ""
org: ""
othername: ""
others: Array []
pg: Object {  }
pgname: ""
photo: Object { types: [], value: "", localURI: "", … }
prefixname: ""
prodid: "-//Thunderbird.net/NONSGML Thunderbird CardBook V28.8//EN"
rev: "20180511T195945Z"
role: ""
sortstring: ""
sound: Object { types: [], value: "", localURI: "", … }
suffixname: ""
tel: Array []
title: ""
tz: ""
uid: "28ff0dbc-1803-44a6-86be-b980ae41fa76"
updated: false
url: Array []
version: "4.0"

[RealRaven2000]

I just did a first check in after writing conversion code for Cardbook formats. Here is a first build that does something worth testing:

smartTemplate-fx-3.16pre50.zip

Added a special switch that needs to be activated in about:config. Just set

extensions.smartTemplate4.mime.resolveAB.CardBook = true

to activate Cardbook support. You also need the minimal version of 84.4 which implements the prototype version of card lookup which is used to extract the data from the other Add-on:

cardbook-84.4-tb.zip

To install these preview versions, download both zip files and drag them into Thunderbird Add-ons Manager, without extracting their contents. I did turn a lot of functions into async, one major problem that remains is that the LoadIdentity function (which I monkey patched) had to be turned into async - which can lead to problems when it is called by Thunderbird itself - it may potentially execute its own code too early as it does not know that it would need to "await" SmartTemplates to finish its own processing first.

[RealRaven2000]

The changeset for going async: b3206cc
shows which functions have been converted to asynchronous flow.

[RealRaven2000]

I added User Interface integration for Cardbook - find the new option to enable Cardbook on the advanced settings tab in SmartTemplates Settings:

smartTemplate-fx-3.16pre61.zip

UI is already translated to the supported languages. I also added a better error message (and detailed loggin in error console) if any exceptions happen during replacing of SmartTemplates variables.

To install the above preview version download the zip files and drag it into Thunderbird Add-ons Manager, without extracting their contents. Note that you need at least Cardbook 84.4 installed (see version above) to test this new feature.

[RealRaven2000]

A new milestone was reached - with John Bieling's help I fixed the asynchronous replacement code which broke importing other templates via %file()% - Here is the latest version:

smartTemplate-fx-3.16pre72.zip

To install the above preview version download the zip files and drag it into Thunderbird Add-ons Manager, without extracting their contents. Note that you need at least Cardbook 84.4 installed (see version above) to test this new feature.

[tkcantonedu]

Do I need the pro license to use it? I have the standard license.

[RealRaven2000]

Do I need the pro license to use it? I have the standard license.

yes, this will be a premium feature, Pro License (or Domain license) is required. It's a very complex feature which required very fundamental rewriting + there is always the risk of depending on another Add-on and its potential changes.

If you want to wait until it lands officially in the next version 3.16 you can save some money on the upgrade as there will be a substantial bargain offer for upgraders when I launch it.

[RealRaven2000]

Do I need the pro license to use it? I have the standard license.

PS: If you want to test now, you contact me directly for a test license.

[tkcantonedu]

Sounds good to me. I'll wait for the official release. Thanks!

[RealRaven2000]

Here is a test version with some interesting caveats. This parses the Cardbook structure directly. But I also improved the vCard parser (which is the new format used by Thunderbird standard AB) The way I parsed the vCard structure directly wasn't optimal as there are different version numbers which are implemented differently and could lead to failures. In this version I am using the ical library to parse out a jCal (json) structure, that hopefully is version independent. Since the CardBook part is parsing the native structure I will obviously need a heads up by the author in case he wants to modify the internal structure of his card storage.

smartTemplate-fx-3.16pre109.zip

Test template with all supported fields
AB.zip

[RealRaven2000]

Another version - adds improvements in vCard reading and CardBook for organisation / departements. You can now have multiple departments after the Organisation by delimiting them with ; - these will automatically delimited with <br>. I am also contemplating whether to add an option "fallback to standard AB" if no entry is found in Cardbook.

smartTemplate-fx-3.16pre115.zip

To install, download the zip file and drag it into Thunderbird Add-ons Manager.

[RealRaven2000]

New test template:

AB.zip

unpack the html file and add to the "reply" menu (configure menu item) or just open it directly:

[RealRaven2000]

Latest version - added a "fallback to standard AB option" when contact is not found in Cardbook:

smartTemplate-fx-3.16pre126.zip

To try out, download the zip file and drag it into Thunderbird Add-ons Manager (don't extract contents).

[RealRaven2000]

implemented in version 3.16 published on 6th of March 2023

[tkcantonedu]

Hi, I upgraded to the pro version and have used it for several weeks. It works great. Thanks!