JAMES-2910 HTML could be indexed directly in ElasticSearch #2739

chibenwa · 2019-10-03T09:35:21Z

No description provided.

…bled

The size of indexed JSON documents significantly decreases for emails containing HTML (eg from 196 KB to 8.6 KB)

This avoids needless network ops

…en disabled

remk · 2019-10-03T11:54:42Z

...rc/test/java/org/apache/james/mailbox/elasticsearch/json/MessageToElasticSearchJsonTest.java

+                IndexAttachments.NO);
+        String convertToJsonWithoutAttachment = messageToElasticSearchJson.convertToJsonWithoutAttachment(message, ImmutableList.of(USER));
+
+        System.out.println(convertToJsonWithoutAttachment);


mailbox/tika/src/main/java/org/apache/james/mailbox/tika/TikaTextExtractor.java

chibenwa · 2019-10-04T06:03:22Z

[36f2f7d1a8ca2f13e1324b998779ed1df8df0dd8] [INFO] # There is insufficient memory for the Java Runtime Environment to continue.
[36f2f7d1a8ca2f13e1324b998779ed1df8df0dd8] [ERROR] # There is insufficient memory for the Java Runtime Environment to continue.OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000633c00000, 484442112, 0) failed; error='Not enough space' (errno=12)
[36f2f7d1a8ca2f13e1324b998779ed1df8df0dd8] [INFO] # Native memory allocation (mmap) failed to map 484442112 bytes for committing reserved memory.
[36f2f7d1a8ca2f13e1324b998779ed1df8df0dd8] [ERROR] # An error report file with more information is saved as:
[36f2f7d1a8ca2f13e1324b998779ed1df8df0dd8] [INFO] # /james-project/hs_err_pid1041.log

mvn test 🍏

test this please

mailbox/store/src/main/java/org/apache/james/mailbox/store/extractor/JsoupTextExtractor.java

.../guice/cassandra-guice/src/main/java/org/apache/james/modules/mailbox/TikaMailboxModule.java

mailbox/store/src/main/java/org/apache/james/mailbox/store/extractor/JsoupTextExtractor.java

mailbox/tika/src/main/java/org/apache/james/mailbox/tika/TikaTextExtractor.java

mailbox/tika/src/test/java/org/apache/james/mailbox/tika/TikaTextExtractorTest.java

mailbox/store/src/main/java/org/apache/james/mailbox/store/extractor/JsoupTextExtractor.java

…en disabled

chibenwa · 2019-10-04T12:49:37Z

MemorySpamAssassinContractTest.spamShouldBeDeliveredInSpamMailboxOrInboxWhenMultipleRecipientsConfigurations » ConditionTimeout

test this please

mailbox/store/src/main/java/org/apache/james/mailbox/store/extractor/JsoupTextExtractor.java

mailbox/tika/src/test/java/org/apache/james/mailbox/tika/TikaTextExtractorTest.java

…e breaks

chibenwa · 2019-10-07T04:35:50Z

[1c0f97530f51eb950a44300e32f2a748e7a940e5] [ERROR] sendingOneHundredMessagesShouldBeRandomlyAssignedToEveryMailboxesOfEveryUsers  Time elapsed: 323.578 s  <<< ERROR!
[1c0f97530f51eb950a44300e32f2a748e7a940e5] org.awaitility.core.ConditionTimeoutException: 
[1c0f97530f51eb950a44300e32f2a748e7a940e5] Assertion condition defined as a lambda expression in org.apache.james.smtp.SmtpRandomStoringTest 
[1c0f97530f51eb950a44300e32f2a748e7a940e5] Expecting:
[1c0f97530f51eb950a44300e32f2a748e7a940e5]  <39L>
[1c0f97530f51eb950a44300e32f2a748e7a940e5] to be between:
[1c0f97530f51eb950a44300e32f2a748e7a940e5]  [400L, 800L] within 10 seconds.
[1c0f97530f51eb950a44300e32f2a748e7a940e5] 	at org.apache.james.smtp.SmtpRandomStoringTest.sendingOneHundredMessagesShouldBeRandomlyAssignedToEveryMailboxesOfEveryUsers(SmtpRandomStoringTest.java:156)
[1c0f97530f51eb950a44300e32f2a748e7a940e5] Caused by: java.util.concurrent.TimeoutException
[1c0f97530f51eb950a44300e32f2a748e7a940e5] 	at org.apache.james.smtp.SmtpRandomStoringTest.sendingOneHundredMessagesShouldBeRandomlyAssignedToEveryMailboxesOfEveryUsers(SmtpRandomStoringTest.java:156)
[1c0f97530f51eb950a44300e32f2a748e7a940e5]

test this please

mailbox/tika/src/test/java/org/apache/james/mailbox/tika/TikaTextExtractorTest.java

chibenwa · 2019-10-14T02:31:23Z

The email corpus we tested this against does not contains HTML nor attachment and thus is not significative for performance evaluation of this change set.

chibenwa · 2019-10-15T03:22:21Z

Merged

chibenwa added 3 commits October 3, 2019 16:34

JAMES-2910 TikaModule should fallback to JsoupTextExtractor when disa…

6061f7a

…bled

JAMES-2910 Extract text from HTML upon ElasticSearch indexation

06e8936

The size of indexed JSON documents significantly decreases for emails containing HTML (eg from 196 KB to 8.6 KB)

JAMES-2910 Tika should delegate text extraction to Jsoup

75e9964

This avoids needless network ops

chibenwa added the perf Contributes some performance enhencements label Oct 3, 2019

chibenwa added this to the Sprint 12 - Robusta beans milestone Oct 3, 2019

chibenwa added 3 commits October 3, 2019 17:08

fixup! JAMES-2910 TikaModule should fallback to JsoupTextExtractor wh…

93000ee

…en disabled

fixup! JAMES-2910 TikaModule should fallback to JsoupTextExtractor wh…

7f621fd

…en disabled

fixup! JAMES-2910 TikaModule should fallback to JsoupTextExtractor wh…

72c6cc9

…en disabled

remk reviewed Oct 3, 2019

View reviewed changes

remk approved these changes Oct 3, 2019

View reviewed changes

rouazana reviewed Oct 3, 2019

View reviewed changes

mailbox/tika/src/main/java/org/apache/james/mailbox/tika/TikaTextExtractor.java Show resolved Hide resolved

JAMES-2910 Jsoup should also handle all text mediaTypes

36f2f7d

trantienduchn approved these changes Oct 4, 2019

View reviewed changes

remk reviewed Oct 4, 2019

View reviewed changes

mailbox/store/src/main/java/org/apache/james/mailbox/store/extractor/JsoupTextExtractor.java Outdated Show resolved Hide resolved

Arsnael approved these changes Oct 4, 2019

View reviewed changes

mbaechler reviewed Oct 4, 2019

View reviewed changes

chibenwa added 6 commits October 4, 2019 15:50

JAMES-2910 Jsoup is more efficient handling inputStream

7bc4179

fixup! JAMES-2910 TikaModule should fallback to JsoupTextExtractor wh…

b36e6e4

…en disabled

fixup! JAMES-2910 Jsoup should also handle all text mediaTypes

a764a0b

JAMES-2910 Use return early code style within JsoupTextExtractor

73cc6e6

fixup! JAMES-2910 Tika should delegate text extraction to Jsoup

cb82eaf

JAMES-2910 TikaTextExtractorTest shouldn't assert strongly line breaks

6279952

Arsnael approved these changes Oct 4, 2019

View reviewed changes

mbaechler reviewed Oct 4, 2019

View reviewed changes

mailbox/store/src/main/java/org/apache/james/mailbox/store/extractor/JsoupTextExtractor.java Show resolved Hide resolved

mailbox/tika/src/test/java/org/apache/james/mailbox/tika/TikaTextExtractorTest.java Outdated Show resolved Hide resolved

chibenwa added 2 commits October 7, 2019 09:42

fixup! JAMES-2910 Use return early code style within JsoupTextExtractor

e97c01f

fixup! JAMES-2910 TikaTextExtractorTest shouldn't assert strongly lin…

1c0f975

…e breaks

Arsnael approved these changes Oct 7, 2019

View reviewed changes

remk reviewed Oct 7, 2019

View reviewed changes

mailbox/tika/src/test/java/org/apache/james/mailbox/tika/TikaTextExtractorTest.java Show resolved Hide resolved

remk approved these changes Oct 7, 2019

View reviewed changes

mbaechler approved these changes Oct 7, 2019

View reviewed changes

chibenwa closed this Oct 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JAMES-2910 HTML could be indexed directly in ElasticSearch #2739

JAMES-2910 HTML could be indexed directly in ElasticSearch #2739

chibenwa commented Oct 3, 2019

remk Oct 3, 2019

chibenwa commented Oct 4, 2019

chibenwa commented Oct 4, 2019

chibenwa commented Oct 7, 2019

chibenwa commented Oct 14, 2019

chibenwa commented Oct 15, 2019

JAMES-2910 HTML could be indexed directly in ElasticSearch #2739

JAMES-2910 HTML could be indexed directly in ElasticSearch #2739

Conversation

chibenwa commented Oct 3, 2019

remk Oct 3, 2019

Choose a reason for hiding this comment

chibenwa commented Oct 4, 2019

chibenwa commented Oct 4, 2019

chibenwa commented Oct 7, 2019

chibenwa commented Oct 14, 2019

chibenwa commented Oct 15, 2019