DNS MX record not used for mailserver lookup and failback

[vogthenn]

Describe the bug
at a client we have a mail server that has only a registered MX record, with mulitple servers coded in there. The designated mail server has no other DNS record, though, and lookup fails with "UnknownHostException":
com.sun.mail.util.MailConnectException: Couldn't connect to host, port: smtpUAT.mail.mycustomer.com, 25; timeout -1;
nested exception is:
java.net.UnknownHostException: smtpUAT.mail.mycustomer.com

I adapted the smtp mailserver slightly.
Here the lookups:
$ nslookup smtpUAT.mail.mycustomer.com
Server: 10.255.253.1
Address: 10.255.253.1#53

*** Can't find smtpUAT.mail.mycustomer.com: No answer

$ nslookup -type=mx smtpUAT.mail.mycustomer.com
Server: 10.255.253.1
Address: 10.255.253.1#53

smtpUAT.mail.mycustomer.com mail exchanger = 10 mailhost1.mycustomer.com.
smtpUAT.mail.mycustomer.com mail exchanger = 10 mailhost2.mycustomer.com.
smtpUAT.mail.mycustomer.com mail exchanger = 10 mailhost3.mycustomer.com.
smtpUAT.mail.mycustomer.com mail exchanger = 10 mailhost4.mycustomer.com.

To Reproduce
Steps to reproduce the behavior:

1. Create an MX record pointing to an arbitrary, existing mail server (smtp.googlemail.com, for example)
2. Use this server in an arbitrary mail sending test
3. See error:
   com.sun.mail.util.MailConnectException: Couldn't connect to host, port: smtpUAT.mail.mycustomer.com, 25; timeout -1;
   nested exception is:
   java.net.UnknownHostException: smtpUAT.mail.mycustomer.com

Expected behavior
The RFC states, that after MX record looup, the CNAME should be used. Jakarta mail uses the standard java DNS lookup, and does not bother to do the slightly different MX record lookup, with fallback servers, if something goes wrong.

Screenshots
N/A

Desktop (please complete the following information):

• OS: Windows/Linux (not relevant)
• Browser: None, used a unit-test
• Version jakarta mail 2.0.1

Smartphone (please complete the following information):
N/A

Mail server:

• Protocol being used: smtp
• Vendor/product: unkown
• Mail service URL: internal (but should be testable with own DNS server)

Additional context
The expecteation would be

• MX Records are used for lookup
• If connection fails, backups are used
• If MX lookup fails, default java is used

I found this snippet on how to lookup MX records in Java:
static String[] lookupMailHosts(String domainName) throws NamingException {
// see: RFC 974 - Mail routing and the domain system
// see: RFC 1034 - Domain names - concepts and facilities
// see: http://java.sun.com/j2se/1.5.0/docs/guide/jndi/jndi-dns.html
// - DNS Service Provider for the Java Naming Directory Interface (JNDI)

// get the default initial Directory Context
InitialDirContext iDirC = new InitialDirContext();
// get the MX records from the default DNS directory service provider
//    NamingException thrown if no DNS record found for domainName
Attributes attributes = iDirC.getAttributes("dns:/" + domainName, new String[] { "MX" });
// attributeMX is an attribute ('list') of the Mail Exchange(MX) Resource Records(RR)
Attribute attributeMX = attributes.get("MX");

// if there are no MX RRs then default to domainName (see: RFC 974)
if (attributeMX == null) {
  return (new String[] { domainName });
}

// split MX RRs into Preference Values(pvhn[0]) and Host Names(pvhn[1])
String[][] pvhn = new String[attributeMX.size()][2];
for (int i = 0; i < attributeMX.size(); i++) {
  pvhn[i] = ("" + attributeMX.get(i)).split("\\s+");
}

// sort the MX RRs by RR value (lower is preferred)
Arrays.sort(pvhn, new Comparator<String[]>() {

  public int compare(String[] o1, String[] o2) {
    return (Integer.parseInt(o1[0]) - Integer.parseInt(o2[0]));
  }
});

// put sorted host names in an array, get rid of any trailing '.'
String[] sortedHostNames = new String[pvhn.length];
for (int i = 0; i < pvhn.length; i++) {
  sortedHostNames[i] = pvhn[i][1].endsWith(".") ? pvhn[i][1].substring(0, pvhn[i][1].length() - 1) : pvhn[i][1];
}
return sortedHostNames;

}

[leonardwoo]

please support custom naming server

[jmehrens]

Interesting. I think JEP 418: Internet-Address Resolution SPI shipped with JDK 18 might be an option to get the existing code working the way you want it to work.

My other thought is that you have provided the JNDI exception to perform the MX lookup so why not layer this code in your application? Just do the MX lookup and create a list of session objects that differ by host name. Building on your example:

Properties parent = new Properties(); //All common properties
Session[] sessions = new Session[sortedHostNames]; //From the MX lookup
for (int i=0; i<sortedHostNames.length; i++) {
    Properties child = new Properties(parent);
    child.put("mail.host", sortedHostNames[i]);
    //set port for child???
    sessions[i] = Session.getInstance(child);
}

for (Session s : sessions) {
    try (Transport t = s.getTransport()) {
          t.connect();
          return s; //Return active session or service
    } catch(MessagingException skip) {
    }
}
return null; //All sites are down.

//Use returned session to build and transport your message.

I suppose the blocker would be abstractions over JakartaMail wouldn't support this behavior.

Few things that have to be dealt with internally with the project are:

1. JakartaMail doesn't have a dependency on java.naming which would need to be addressed to proceed on this.
2. Doing a MX lookup and reducing that list to one entry would be viable but, implementing that fallback logic as described in description is too complicated without SMTP pooling support.
3. Host name lookups are used to compute some headers like Message.setFrom with no arguments. Not sure if MX lookup would impact things like that in the codebase. More review is needed there.
4. Should we create a JEP 418 InetAddressResolverProvider implementation in JakaraMail / AngusMail as its own artifact or at all?