Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Email 2.2 #11554

Merged
merged 15 commits into from
Jul 29, 2021
Merged

Email 2.2 #11554

merged 15 commits into from
Jul 29, 2021

Conversation

StorytellerCZ
Copy link
Collaborator

@StorytellerCZ StorytellerCZ commented Jul 22, 2021

Implements new features as per: #11547

@StorytellerCZ
Copy link
Collaborator Author

@xet7 what do you think?

@StorytellerCZ StorytellerCZ marked this pull request as ready for review July 22, 2021 09:54
@harryadel
Copy link
Contributor

Isn't it a good a time to remove the fibers stuff, convert to async and do a major bump?

@StorytellerCZ
Copy link
Collaborator Author

@harryadel Let's leave that for the next version. I think these changes are enough for now.

@diavrank
Copy link
Contributor

This feature is awesome. I was wondering if it also consider to add more than one transport in order to contemplate this feature request.

@xet7
Copy link
Contributor

xet7 commented Jul 23, 2021

@xet7 what do you think?

I'll try some way way to test this. I tried:

  • Using meteor fork of this PR, but Wekan is not yet upgraded to newest Meteor
  • Making fork of only email package, but it seems other packages depend on email@2.0.0 package

Maybe I could first build current Wekan bundle, and then replace email.js etc in that bundle, and run that bundle. I'll try that next.

@StorytellerCZ
Copy link
Collaborator Author

@xet7 I would recommend just copying it into the local packages directory and then you can adjust the version as needed to make it work.

@xet7
Copy link
Contributor

xet7 commented Jul 24, 2021

@StorytellerCZ

@xet7 I would recommend just copying it into the local packages directory and then you can adjust the version as needed to make it work.

I could not get it working in packages directory, so I copied code from email.js to wekan/.build/bundle/programs/server/packages/email.js.

Then I did some changes, to port it Node 12.x and Meteor 2.2 that Wekan currently used.

Here is changed email.js in .zip file: email.zip

Here are changes I made:

export const
=>
const

const host = url?.split(':')[0];
=>
const host = url.split(':')[0];

  const transport = nodemailer.createTransport({
    service: settings?.service || service,
    auth: {
      user: settings?.user || user,
      pass: settings?.password || password
    }
  });
=>
  const transport = nodemailer.createTransport({
    service: settings.service || service,
    auth: {
      user: settings.user || user,
      pass: settings.password || password
    }
  });

const packageSettings = Meteor.settings.packages?.email || {};
=>
const packageSettings = Meteor.settings.packages.email || {};

if ((packageSettings?.service && knownHosts.includes(packageSettings)) || knownHosts.includes(url?.split(':')[0])) {
=>
if ((packageSettings.service && knownHosts.includes(packageSettings)) || knownHosts.includes(url.split(':')[0])) {

// New Meteor code
//const sendHooks = new Hook();
// Old Meteor code backport:
const sendHooks = [];

Email.hookSend = function (f) {
  // New Meteor code:
  //sendHooks.register(f);
  // Old Meteor code backport:
  sendHooks.push(f);
};

But now that I try to invite new user to Wekan board with user's email address, that would send email for invite, I get this error:

Exception while invoking method 'inviteUserToBoard' TypeError: Cannot read property 'send' of undefined
    at AccountsServer.Accounts.sendEnrollmentEmail (packages/accounts-password/password_server.js:804:9)
    at MethodInvocation.inviteUserToBoard (models/users.js:1196:18)
    at packages/check/match.js:118:15
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:18)
    at packages/ddp-server/livedata_server.js:719:19
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at packages/ddp-server/livedata_server.js:717:46
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at packages/ddp-server/livedata_server.js:715:46
    at new Promise (<anonymous>)
    at Session.method (packages/ddp-server/livedata_server.js:689:23)
    at packages/ddp-server/livedata_server.js:559:43

So I think some of the changes have not been ported correctly think.

Because I could not get this email package working yet with , I think next I'll try:

a) Figure out minimal changes to current Meteor 2.2 email package that would still work with Node 12.x, Meteor 2.2 and current Wekan.

b) Creating new example project with newest Meteor and this email package, to test does sending email work with newest Meteor.

@StorytellerCZ
Copy link
Collaborator Author

Or this might be related to the test failures that we see in CI.

Fix new tests for email by making sure that they clean after themselves and allow other tests to run.
Fix included search for known services.
Return hook from hook registration, so that it can be stopped.
@StorytellerCZ
Copy link
Collaborator Author

Alright! I have fixed the tests. The issue there was that they weren't properly cleaning after themselves.

@xet7 I think you will need to make changes in package.js as well to get it working. The package is now using the new syntax there which is why you are getting the undefined error.

@xet7
Copy link
Contributor

xet7 commented Jul 26, 2021

@StorytellerCZ

Thanks! I'll try newest changes.

@xet7
Copy link
Contributor

xet7 commented Jul 26, 2021

@StorytellerCZ

I don't understand why this does not work:

export MAIL_URL=Outlook365://firstname.lastname%40hotmail.com:password@hotmail.com

I get error:

(Mail not sent; to enable sending, set the MAIL_URL environment variable.)

But in beginning of start-wekan.sh I have:

echo "MAIL_URL = $MAIL_URL"

And it shows that MAIL_URL I did set. Do I need to make some other code changes?

To package.js I did add name and current version for compatibility:

Package.describe({
  name: "email",
  summary: "Send email messages",
  version: "2.0.0"
});

//  Actual version. Above 2.0.0 only for compatibility reasons.
//  version: "2.2.0-beta240.1"

And then built wekan with ./rebuild-wekan.sh.

I did also verify that new email.js code is at wekan/.build/bundle/programs/server/packages/email.js.

@StorytellerCZ
Copy link
Collaborator Author

@xet7 You will need the Package.onUse section most importantly, but in this case what is messing it up is the custom protocol which then means that URL does not recognize the username and password. I think it might be better to switch to automatically recognize known hosts based on hostname rather than protocol.

@xet7
Copy link
Contributor

xet7 commented Jul 26, 2021

@StorytellerCZ

@xet7 You will need the Package.onUse section most importantly, but in this case what is messing it up is the custom protocol which then means that URL does not recognize the username and password. I think it might be better to switch to automatically recognize known hosts based on hostname rather than protocol.

I did use all of new email package, including package.js new code.

How would known hosts work? In this case, it's detected from protocol:

export MAIL_URL=Outlook365://firstname.lastname%40hotmail.com:password@hotmail.com

But usually at O356, Google Workplace, etc companies have custom domain:

export MAIL_URL=Outlook365://firstname.lastname%40yourcompany.com:password@yourcompany.com

If it were just smtp, it's hard to detect, because protocol=service is not mentioned anywhere:

export MAIL_URL=smtp://firstname.lastname%40yourcompany.com:password@yourcompany.com

Or should service be option?

export MAIL_URL=smtp://firstname.lastname%40yourcompany.com:password@yourcompany.com?service=Outlook365

@StorytellerCZ
Copy link
Collaborator Author

StorytellerCZ commented Jul 27, 2021

Custom domains would not work with know-services from what I can tell as it only links to the main domain. So in that case it would be useless. Known services takes only login and password and then builds the rest of the url from hardcoded settings.

@xet7
Copy link
Contributor

xet7 commented Jul 27, 2021

@StorytellerCZ

With what kind of code I could send email with Outlook365 service and your newest email package code?

@StorytellerCZ
Copy link
Collaborator Author

If we look at, for example Gmail, this is what the well-know services knows:

"Gmail": {
        "aliases": ["Google Mail"],
        "domains": ["gmail.com", "googlemail.com"],
        "host": "smtp.gmail.com",
        "port": 465,
        "secure": true
    },

So upon detecting Gmail it then sets the host, port and secure connection with the transport and we'll add in the login and password. I'll try to improve the detection of the service a bit more today.
Correct me if I'm wrong, but shouldn't the login for say Gmail also include the custom domain. IE. storyteller@meteor.com login send to Gmail services would resolve to the Meteor company account?

@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

@StorytellerCZ

With your latest changes, with what kind of URL syntax does custom service email sending work?

@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

@StorytellerCZ

When I tested yesterday, it still did not detect that MAIL_URL is set at all.

@StorytellerCZ
Copy link
Collaborator Author

The package settings takes precedent. Here is the logic:

const getTransport = function() {
  const packageSettings = Meteor.settings.packages?.email || {};
  // We delay this check until the first call to Email.send, in case someone
  // set process.env.MAIL_URL in startup code. Then we store in a cache until
  // process.env.MAIL_URL changes.
  const url = process.env.MAIL_URL;
  if (this.cacheKey === undefined || (this.cacheKey !== url || this.cacheKey !== packageSettings?.service || 'settings')) {
    if ((packageSettings?.service && wellKnow(packageSettings.service)) || (url && wellKnow(new URL(url).hostname) || wellKnow(url?.split(':')[0] || ''))) {
      this.cacheKey = packageSettings.service || 'settings';
      this.cache = knownHostsTransport(packageSettings, url);
    } else {
      this.cacheKey = url;
      this.cache = url ? makeTransport(url, packageSettings) : null;
    }
  }
  return this.cache;
};

@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

@StorytellerCZ

What kind of code is required to send email?

@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

It seems to me that MAIL_URL is not detected at all.

@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

Anyway, I'll test with todays changes, just a moment.

@StorytellerCZ
Copy link
Collaborator Author

Did a complete test locally and fixed the bug you were referring to @xet7
Merging now and releasing new 2.4 beta to get more broader testing.

@StorytellerCZ StorytellerCZ merged commit 4b7f121 into release-2.4 Jul 29, 2021
@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

@StorytellerCZ

Thanks a lot! I'll test :)

@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

@StorytellerCZ

Hmm, interesting error, I'll try to figure it out.

> Starting add-description-text-allowed migration.
> Finishing add-description-text-allowed migration.
> Starting add-sort-field-to-boards migration.
> Finishing add-sort-field-to-boards migration.
> Starting add-default-profile-view migration.
> Finishing add-default-profile-view migration.
> Starting add-hide-logo-by-default migration.
> Finishing add-hide-logo-by-default migration.
{"line":"87","file":"percolate_synced-cron.js","message":"SyncedCron: Scheduled \"notification_cleanup\" next run @Fri Jul 30 2021 00:03:08 GMT+0300 (GMT+03:00)","time":{"$date":1627592588955},"level":"info"}
Exception while invoking method 'ATCreateUserServer' TypeError: Cannot read property 'split' of undefined
    at module.exports (/home/wekan/repos/wekan/.build/bundle/programs/server/npm/node_modules/meteor/email/node_modules/nodemailer/lib/well-known/index.js:45:28)
    at knownHostsTransport (packages/email/email.js:81:8)
    at getTransport (packages/email/email.js:107:20)
    at Object.Email.send (packages/email/email.js:221:47)
    at AccountsServer.Accounts.sendVerificationEmail (packages/accounts-password/password_server.js:923:9)
    at MethodInvocation.ATCreateUserServer (packages/useraccounts_core.js:1310:16)
    at packages/check/match.js:118:15
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:18)
    at packages/ddp-server/livedata_server.js:719:19
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at packages/ddp-server/livedata_server.js:717:46
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at packages/ddp-server/livedata_server.js:715:46
    at new Promise (<anonymous>)
    at Session.method (packages/ddp-server/livedata_server.js:689:23)
    at packages/ddp-server/livedata_server.js:559:43

@StorytellerCZ StorytellerCZ deleted the feature/email-2-2 branch July 29, 2021 21:07
@xet7
Copy link
Contributor

xet7 commented Jul 29, 2021

@StorytellerCZ

It seems that this line:

at module.exports (/home/wekan/repos/wekan/.build/bundle/programs/server/npm/node_modules/meteor/email/node_modules/nodemailer/lib/well-known/index.js:45:28)

Has this this kind of split code:

module.exports = function (key) {
    key = normalizeKey(key.split('@').pop());
    return normalized[key] || false;
};

I try to figure out why it says:

Exception while invoking method 'ATCreateUserServer' TypeError: Cannot read property 'split' of undefined

@StorytellerCZ
Copy link
Collaborator Author

What kind of URL were you using?

@xet7
Copy link
Contributor

xet7 commented Jul 30, 2021

@StorytellerCZ

export MAIL_URL=Outlook365://firstname.lastname%40hotmail.com:password@hotmail.com

or:

export MAIL_URL=Outlook365://firstname.lastname@hotmail.com:password@hotmail.com

@xet7
Copy link
Contributor

xet7 commented Jul 30, 2021

@StorytellerCZ

And also:

export MAIL_URL=Outlook365://firstname.lastname:password@hotmail.com

All have same split error.

@daxxog
Copy link

daxxog commented Aug 1, 2021

Does this change fix this issue? Basically the "name" value needs to be set to the users domain in order for Google Workspace to accept the SMTP EHLO.

@xet7
Copy link
Contributor

xet7 commented Aug 1, 2021

@daxxog

Maybe. I think I anyway need to write only nodemailer using code to Wekan, to get this working. Also Wekan is still stuck on Meteor 2.2 and Node 12.x, I have not had any success in upgrading.

@xet7
Copy link
Contributor

xet7 commented Aug 1, 2021

I'm not saying that any other web framework or programming language is any better than Meteor, everything else I have tried is even more broken in many ways.

@StorytellerCZ
Copy link
Collaborator Author

@xet7 I will add your examples to the test suite and fix them, but I'm afraid that since URL can't decipher these objects we will constantly be fixing infinite sets of possible url strings. This though is more of a stop gap measure until people can migrate to the settings option which is going to be 100% accurate.

@StorytellerCZ
Copy link
Collaborator Author

@xet7 even with addition of your test cases the tests are still passing for me. I'll do a more complex test later.

@xet7
Copy link
Contributor

xet7 commented Aug 3, 2021

@StorytellerCZ

Thanks! I'll test.

xet7 added a commit to wekan/wekan that referenced this pull request Aug 3, 2021
@xet7
Copy link
Contributor

xet7 commented Aug 3, 2021

@StorytellerCZ

With these examples:

export MAIL_URL=Outlook365://firstname.lastname%40hotmail.com:password@hotmail.com
export MAIL_URL=Outlook365://firstname.lastname@hotmail.com:password@hotmail.com
export MAIL_URL=Hotmail://firstname.lastname@hotmail.com:password@hotmail.com

I still get this error

> Finishing assign-boardwise-card-numbers migration.
{"line":"87","file":"percolate_synced-cron.js","message":"SyncedCron: Scheduled \"notification_cleanup\" next run @Tue Aug 03 2021 19:56:35 GMT+0300 (GMT+03:00)","time":{"$date":1628009795539},"level":"info"}
(node:15795) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Exception while invoking method 'ATCreateUserServer' TypeError: Cannot read property 'split' of undefined
    at module.exports (/home/wekan/repos/wekan/.build/bundle/programs/server/npm/node_modules/meteor/email/node_modules/nodemailer/lib/well-known/index.js:45:28)
    at knownHostsTransport (packages/email/email.js:81:8)
    at getTransport (packages/email/email.js:107:20)
    at Object.Email.send (packages/email/email.js:221:47)
    at AccountsServer.Accounts.sendVerificationEmail (packages/accounts-password/password_server.js:923:9)
    at MethodInvocation.ATCreateUserServer (packages/useraccounts_core.js:1310:16)
    at packages/check/match.js:118:15
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at Object._failIfArgumentsAreNotAllChecked (packages/check/match.js:116:43)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:18)
    at packages/ddp-server/livedata_server.js:719:19
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at packages/ddp-server/livedata_server.js:717:46
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at packages/ddp-server/livedata_server.js:715:46
    at new Promise (<anonymous>)
    at Session.method (packages/ddp-server/livedata_server.js:689:23)
    at packages/ddp-server/livedata_server.js:559:43

That file wekan/.build/bundle/programs/server/npm/node_modules/meteor/email/node_modules/nodemailer/lib/well-known/index.js has this code:

/**
 * Resolves SMTP config for given key. Key can be a name (like 'Gmail'), alias (like 'Google Mail') or
 * an email address (like 'test@googlemail.com').
 *
 * @param {String} key [description]
 * @returns {Object} SMTP config or false if not found
 */
module.exports = function (key) {
    key = normalizeKey(key.split('@').pop());
    return normalized[key] || false;
};

This nodemailer code still works OK:

var nodemailer = require('nodemailer');

let transporter = nodemailer.createTransport({
  service: "Outlook365",
  auth: {
    user: 'firstname.lastname@hotmail.com',
    pass: 'password'
  },
})

let info = transporter.sendMail({
  from: 'Wekan Boards <firstname.lastname@hotmail.com>',
  to: 'me@example.com',
  subject: 'Hi',
  text: 'Hi! Reminder: Meeting tomorrow..',
  html: '<h1>Hi! Reminder: Meeting tomorrow.</h1>'
})

@xet7
Copy link
Contributor

xet7 commented Aug 3, 2021

@StorytellerCZ

Code I tried with Meteor 2.2 and Node 12.22.4 is at:
https://github.com/wekan/wekan/tree/feature-email-2-2

@StorytellerCZ
Copy link
Collaborator Author

@xet7 honestly I don't know what to tell you. I will try a few more things, but so far I was unable to replicate the issue.

@xet7
Copy link
Contributor

xet7 commented Aug 4, 2021

@StorytellerCZ

No problem. Please don't use too much time on this. I will fix this in Wekan code anyway.

Copy link
Collaborator

@filipenevola filipenevola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my requested changes.

packages/email/email.js Show resolved Hide resolved
packages/email/email.js Show resolved Hide resolved
packages/email/email.js Show resolved Hide resolved
packages/email/email.js Show resolved Hide resolved
packages/email/email.js Show resolved Hide resolved
packages/email/email_tests.js Show resolved Hide resolved
packages/email/email_tests.js Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants