Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move template processing into background script #184

Open
RealRaven2000 opened this issue May 18, 2022 · 7 comments
Open

Move template processing into background script #184

RealRaven2000 opened this issue May 18, 2022 · 7 comments
Assignees
Labels

Comments

@RealRaven2000
Copy link
Owner

RealRaven2000 commented May 18, 2022

Motivations:

  • In order to use composeScripts and integration with other mail extensions (not just legacy) all template / string processing and the insertion into Composer need to be moved into the background script. The new code will be done on the current ESR branch (currently 91, soon to be 102).
  • Once the main parser runs within background we are an essential step closer to conversion to a native API mail extension [mx] without legacy code and without requiring full permissions to the user's PC. However it is likely that things like UI integration (3 buttons in composer toolbar), full access to external files, access to certain MIME headers etc. might then still not possible with non-experimental Add-ons

Methods:

The main flow should be in an async function as part of the background processing.
Legacy function may be called using the notification API.
Whatever functions can be converted to be 100% mx compatible will be rewriten

For the moment the code will not be used until functionality is complete. For testing, the switch "extensions.smartTemplate4.API" can be used to activate the new flow and will completely disable the legacy flow; there is a temporary UI for this:

image

@RealRaven2000 RealRaven2000 self-assigned this May 18, 2022
@RealRaven2000
Copy link
Owner Author

For enabling the new flow (which will switch off all legacy processing of templates and go the new route) we will be using the preference extensions.smartTemplate4.BackgroundParser. In the legacy part this setting can be read using SmartTemplate4.Preferences.isBackgroundParser()

New mx-compatible versions of legacy modules such smartTemplate-util.js will be created in the "script" subfolder and included from the background script or st-background.html, creating a new mx-global namespace "SmartTemplates" (which will replace the old legacy namespace SmartTemplate4).

@RealRaven2000
Copy link
Owner Author

RealRaven2000 commented May 18, 2022

List of affected functions that directly are called from composing events and trigger template processing:

/* event driven functions */
SmartTemplate4.fileTemplates.onExternalMailProcess(data, composeType) // triggered by external Add-ons such as FiltaQuilla
SmartTemplate4.classSmartTemplate.insertTemplate(startup, flags, fileTemplateSource) // main switch using  gMsgCompose.type
SmartTemplate4.notifyComposeBodyReady(evt, isChangeTemplate, win=null) // in smartTemplate.main, calls insertTemplate
SmartTemplate4.loadIdentity(startup, previousIdentity) // calls insertTemplate
SmartTemplate4.fileTemplates.insertFileEntryInComposer(entry) // insert HTML snippet from menu
SmartTemplate4.fileTemplates.onSelectAdHoc(fileTemplateInstance, composeType, popup, btn, singleMsgWindow) // select a html file interactively to change / apply template

/* utility functions */
SmartTemplate4.smartTemplate // this is an instance of SmartTemplate4.classSmartTemplate. plan to abolish this architecture!
SmartTemplate4.smartTemplate.getProcessedText()  // in smartTemplate-compose.js
SmartTemplate4.smartTemplate.extractSignature()   // in smartTemplate-compose.js
SmartTemplate4.classSmartTemplate.getQuoteHeader(composeType, idKey)  // in smartTemplate-compose.js

/* important structures / objects */
SmartTemplate4.pref // interface object for getting legacy account templates based in account.id (key)
SmartTemplate4.PreprocessingFlags // struct to remember state in each individual composer window,
// This also holds a number of semaphores such as NotifyComposeBodyReadyFired, isLoadIdentity that prevent unintentional double processing while streaming files
SmartTemplate4.fileTemplates.armedEntry //  main file template when opening composer, copied from main window instance
SmartTemplate4.fileTemplates.armedQueue // for nesting included files 

@RealRaven2000
Copy link
Owner Author

John recommended to use ESR modules. this also means circular references need to be avoided.
new modules created:

st-compose.mjs.js
st-composer.mjs.js
st-crypto.mjs.js
st-main.mjs.js
st-overlay.mjs.js
st-prefs.mjs.js
st-process.mjs.js
st-signature.mjs.js
st-styles.mjs.js
st-util.mjs.js

1st Problem: SmartTemplates.pref = new SmartTemplates.classPref();
classPref was defined in st-overlay.mjs.js but is needed in various other modules:
It is used in st-compose.mjs.js , st-overlay.mjs.js , st-util.mjs.js. To avoid circular referencing I might have to move this class (or make an instance SmartTemplates.pref) in st-prefs.mjs.js.

st-prefs.mjs.js had references to util, which I removed, as I believe it is used in a lot of places. It currently uses LegacyPrefs and I turned all methods async.

@RealRaven2000
Copy link
Owner Author

latest committed changes:

  • renamed st-overlay.mjs.js to st-parser.mjs.js and added a Parser class as container. Might be converted to a singleton later.
  • messageCompose listener now calls parser.getprocessedText() directly (to be reviewed. it used to be called by insertTemplate) - the parser functions should be completely independent from UI / composer, so anything that needs known about composer state needs to be passed in through the info parameter (an instance of the composers Map)
  • for the moment we will not convert insertTemplate() and just "dump" all changes into composer via messenger.compose.setComposeDetails - there is probably a bunch of logic missing from the 900 lines long function we will need to integrate later. st-compose.mjs.js will deal with composer specific stuff (and may become obsolete in the best case)
  • added a (temporary) checkbox for toggling to the new parser (will disable most legacy handling in composer)

image

@RealRaven2000
Copy link
Owner Author

RealRaven2000 commented May 23, 2022

Weekly Meeting with Add-on Developers - Problem points for conversion

Parser Specific

  • Signature is currently not read - I discovered some problems with reading signature files which is currently not supported by the API
  • composeDetails.type is missing granularity (the cases template, forward as attachment vs inline, MailToUrl are missing or cannot be determined). MailToUrl is an important case as it can be used to transmit data via a query string
  • saving / modifying global state: info.offsets, flags, flags.filePaths[] array, SmartTemplate4.fileTemplates.armedQueue.
  • calendar: uses a passed in parameter that is able to force a different locale (than the one currently used) in order to look up day / month names - I intend to solve this with a call to legacy through background script until the API supports something like this. I am currently still using string bundles in the legacy code in order to support this
  • rebuilding the body within the composer module - there is still a lot of logic around deconstructing and rebuilding the email
  • "dirty" flag in composer. this definitely needs to be reset - gMsgCompose.editor.resetModificationCount() see smartTemplate-compose.js:1739 function resetDocument(editor, withUndo)
  • evalInSandbox - I assume this isn't allowed in a [mx] Add-on?
  • how to retrieve the charset of the replied to email?

Composer Specific

  • late resolved variables. For now I am planning to add these via the legacy code.
  • @jobisoft mentioned that we may need to convert the body into a DOM node for better manipulation. What function should be used? We cannot just overwrite composeDetails.body at the moment as we are losing quoted material and quote header / signature. This was basically code written in smartTemplate-compose.js within the function insertTemplate( ).
    A: use a DOM Parser

@RealRaven2000
Copy link
Owner Author

RealRaven2000 commented May 25, 2022

Work In Progress

  • added composerInfo (info) object to Parser( ) constructor
  • turned a lot of functions async in order to support recursion / signature inserting
  • Started moving Calendar function to Util (this still needs to be finished and l10n can be done internally through a calendar ESR module containing all necessary strings in order to explicitely pass a language parameter (which is currently unsupported by messenger.i18n) - TO DO - complete this task
  • ported some improvements from legacy part (Various Issues in bracketMail parameter in %to% / %from% etc. #186, Support transferring headers / variable results to clipboard  #187 currently WIP)
  • added async version of clipboardWrite() to Util module

Testing

I can currently run a working template from SmartTemplates account settings:

<p>%dateformat(e/n/Y, H:M)%</p>
<p>About: %subject(2)%</p>
<p>Dear %to(firstname)%.<br></p>

<img src="file:///W:/_Tb Profiles/aj1a94hh.axel/stationery/st4-icon.png">

<p>yours,<br>
&nbsp;&nbsp;%from(firstname)%</p>
<hr>
%sig(html)%

Images are currently not fully imported and spawn the content warning to unblock the file (every time). Signature is inserted as expected and also parsed correctly (same problem with contained images). Also the quoted part of the email and quote header are lost as I haven't ported composer integration as such.

@RealRaven2000
Copy link
Owner Author

RealRaven2000 commented Jun 25, 2022

Implemented Util.setSpellchecker() which is API based and can set multiple languages or switch off all dictionaries. Example: %spellcheck(de,en)% will switch on de-DE and en-US dictionaries (if installed). %spellcheck(off)% disables proofing.

smartTemplate-fx-3.12pre74.zip

...enable mx parser in advanced settings to test!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant