Feature attachments #48

pemontto · 2019-09-25T14:17:25Z

Fixes #34

This PR adds a couple of features:

It recursively searches the message parts for attachment and inline attachment filenames
If save_attachments is true it will dump the encoded (usually base64) contents as well

Example content:

{
    ...
    "attachments": [
        {
            "filename": "test-document1.txt",
            "data": "MTI3LjAuMC4x\r\n"
        },
        {
            "filename": "test-attachment2.txt",
            "data": "dGhpcyBpcyBoZWxsbyB3b3JsZAo=\r\n"
        }
    ],
    ...
}

The attachment data can then be used by the Elasticsearch Ingest Attachment Processor Plugin.

Something like:

PUT _ingest/pipeline/test
{
  "description": "Extract attachment information from arrays",
  "processors": [{
      "foreach": {
        "field": "attachments",
        "processor": {
          "gsub": {
            "field": "_ingest._value.data",
            "pattern": "\\s",
            "replacement": ""
          }
        }
      }
    },
    {
      "foreach": {
        "field": "attachments",
        "processor": {
          "attachment": {
            "field": "_ingest._value.data",
            "target_field": "_ingest._value.attachment"
          }
        }
      }
    },
    {
      "foreach": {
        "field": "attachments",
        "processor": {
          "remove": {
            "field": "_ingest._value.data"
          }
        }
      }
    }
  ]
}

pemontto · 2019-09-26T09:52:08Z

Added tests, removed the recursive search and replace it with the inbuilt all_parts method.

pemontto · 2019-10-18T18:12:22Z

@robbavey @karenzone would you be able to review this, #45 and #49?

robbavey

Thanks for the contribution! Couple of small nitpicks and a concern on a potential docs/functionality mismatch - an attachment may not necessarily be base64 encoded, which is inconsistent with the doc.

spec/inputs/imap_spec.rb

lib/logstash/inputs/imap.rb

pemontto

All suggestions reviewed and fixed

karenzone

Thanks, @pemontto. One suggestion: Our documentation standard is to alphabetize config options. Please move save_attachments to above secure in the table and the detailed descriptions.

Otherwise, the docs build cleanly and render correctly, so LGTM. Thanks for adding quality docs along with your code changes.

pemontto

🤦‍♂️fixed the alphabetical ordering

sakthiskv · 2020-05-19T13:00:44Z

I tried above save_attachments , but for me is not working

my conf file

input {
beats {
port => 5044
}
imap {
host => "imap.gmail.com"
user => "sakthiskv@gmail.com"
password => "1234567976736224"
secure => true
check_interval => 3
strip_attachments => "true"
fetch_count => 10000
save_attachments => true
# folder => "MyAwesomeAppEmails" # This line will only work if you apply the above mentioned patch
}
}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "sakthi"
}
}

pemontto · 2020-05-19T13:03:57Z

strip_attachments => "true"

This is likely the cause, you're removing attachments.

sakthiskv · 2020-05-19T13:12:57Z

Thanks for quick reply ,
Its working now , but i am getting .txt files only.
i am not getting .xslx files and images

pemontto · 2020-05-19T13:31:29Z

Are you only seeing the filename without the data field, or is the data field empty? Can you describe in more detail your setup, are you outputting to ES?

sakthiskv · 2020-05-19T14:07:41Z

Yes i am using Es.
I didn't get data into Es, its failing logstash itself. i am getting below error.

[2020-05-19T18:43:49,509][ERROR][logstash.inputs.imap ][main] Encountered error NoMethodError {:message=>"Can not decode an entire message, try calling #decoded on the various fields and body or parts if it is a multipart message.", :backtrace=>["C:/logstash/vendor/bundle/jruby/2.5.0/gems/mail-2.6.6/lib/mail/message.rb:1903:in decoded'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:176:in parse_mail'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:119:in block in check_mail'", "org/jruby/RubyArray.java:1800:in each'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:113:in block in check_mail'", "org/jruby/RubyArray.java:1842:in each_slice'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:111:in check_mail'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:92:in block in run'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in interval'", "C:/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:91:in run'", "C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:314:in inputworker'", "C:/logstash/logstash-core/lib/logstash/java_pipeline.rb:306:in block in start_input'"]}

pemontto · 2020-05-19T16:07:08Z

@sakthiskv thanks for taking the time to run this. Would you be able to send me a copy of the email, or something similar to pemontto@gmail.com so I can take a look? Otherwise we can continue discussion at https://discuss.elastic.co/u/pemontto.

pemontto · 2020-05-19T17:14:03Z

@sakthiskv the code path causing this error isn't actually related to this PR. Would still be interested in seeing the email that causes this.

karenzone

Docs are looking good. One minor request in line.

karenzone · 2020-05-19T17:45:51Z

docs/index.asciidoc

@@ -45,6 +45,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|Yes
 | <<plugins-{type}s-{plugin}-port>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-secure>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-save_attachments>> |<<boolean,boolean>>|No


Docs are building cleanly and generally look good. Will you please move the option in the table to alpha order, too?

@robbavey Will you please review and comment on recent code changes? Thanks!

sakthiskv · 2020-05-20T05:02:28Z

@sakthiskv thanks for taking the time to run this. Would you be able to send me a copy of the email, or something similar to pemontto@gmail.com so I can take a look? Otherwise we can continue discussion at https://discuss.elastic.co/u/pemontto.

I sent mail to above account .
Plz let me know after fixing a problem.

sakthiskv · 2020-05-20T11:59:40Z

[ERROR][logstash.inputs.imap ][main] Encountered error NoMethodError {:message=>"Can not decode an entire message, try calling #decoded on the various fields and body or parts if it is a multipart message.", :backtrace=>["C:/logstash/vendor/bundle/jruby/2.5.0/gems/mail-2.6.6/lib/mail/message.rb:1903:in

I am getting this error. But somehow yesterday it's worked for to get attachment like .txt files .
But now it's failing all the time. Plz help me

thomhubers · 2020-05-27T14:12:11Z

This PR would be highly beneficial for a use case for us too.

robbavey

LGTM - @pemontto Thank you for contribution! I'm happy to merge and publish this if you are satisfied that the issue presented by @sakthiskv is not related to this PR.

pemontto · 2020-05-27T14:23:38Z

Thanks @robbavey, we're debugging offline, the issue is not related to this PR.

thomhubers · 2020-05-27T17:03:42Z

If this is merged, when will it be in a release?

Edit; don't mean to push but we're just really helped here when we can use attachments so eager to test it :)

robbavey · 2020-05-27T17:34:57Z

@thomhubers Once the fix is merged, we will publish the plugin, at which point we will update this issue with the information. From there, the plugin can be manually updated following the instructions here.

karenzone

@pemontto Thanks for your contribution. This is a nice addition!
Congrats on being a first-time contributor. I hope we'll be seeing you again.

Bumps version to publish changes from #48 Reorders options table in documentation

Alsatea · 2020-08-10T17:01:07Z

Thank you for the amazing addition. Is it possible to apply it on .eml email file format? I usually receive emails that have another email attached to it.

yuliansen · 2020-10-08T03:49:06Z

Hi good day for you and everyone. So I would like to try this plugin but problem is my attachments seems to be encrypted when i try to set secure as false its not showing any output, but when i set secure as true, it show the right output even my attachment being encrypted. here is my logstash conf

input {
        imap {
                host => "host"
                password => "password"
                user => "email.co.id"
                folder => "folder"
                port => "993"
                save_attachments=> true
                secure => false
                uid_tracking => false
                check_interval =>10
                fetch_count => 1
         }
}
output
{
stdout{codec=>rubydebug}
}
` ``

pemontto added 5 commits September 25, 2019 15:04

Save attachment filenames and content

d9cc337

Update docs

8794e40

Fix hash syntax for ruby 1.7

a92e5c7

Add test and improve extraction

28e315a

Remove errant puts in spec

99b10fe

pemontto added 3 commits September 26, 2019 11:19

Fix conditional for non multi-part messages

f83f0ef

Fix conditional

60ce689

Simplify attachments loop

aebe1fa

pemontto mentioned this pull request Oct 8, 2019

Attachment Saving - Feature Request #29

Closed

karenzone requested review from robbavey and karenzone October 18, 2019 19:12

robbavey requested changes Nov 18, 2019

View reviewed changes

spec/inputs/imap_spec.rb Outdated Show resolved Hide resolved

lib/logstash/inputs/imap.rb Show resolved Hide resolved

lib/logstash/inputs/imap.rb Outdated Show resolved Hide resolved

Test for unencoded attachment data and update docs

346eb0d

pemontto commented Apr 10, 2020

View reviewed changes

karenzone reviewed Apr 24, 2020

View reviewed changes

karenzone requested a review from robbavey April 24, 2020 17:27

📚Fix alphabetical order in documentation

d1c8e14

pemontto commented Apr 25, 2020

View reviewed changes

karenzone reviewed May 19, 2020

View reviewed changes

robbavey approved these changes May 27, 2020

View reviewed changes

karenzone self-requested a review May 27, 2020 18:19

karenzone approved these changes May 27, 2020

View reviewed changes

karenzone merged commit f35f6ba into logstash-plugins:master May 27, 2020

karenzone mentioned this pull request May 27, 2020

Bump to v.3.1.0 #51

Merged

karenzone added a commit that referenced this pull request May 27, 2020

Bump to v.3.1.0 (#51)

d59ecd1

Bumps version to publish changes from #48 Reorders options table in documentation

thomhubers mentioned this pull request May 29, 2020

Improve method of searching for message body content #49

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature attachments #48

Feature attachments #48

pemontto commented Sep 25, 2019

pemontto commented Sep 26, 2019

pemontto commented Oct 18, 2019

robbavey left a comment

pemontto left a comment

karenzone left a comment

pemontto left a comment

sakthiskv commented May 19, 2020

pemontto commented May 19, 2020

sakthiskv commented May 19, 2020 •

edited

pemontto commented May 19, 2020

sakthiskv commented May 19, 2020

pemontto commented May 19, 2020

pemontto commented May 19, 2020

karenzone left a comment

karenzone May 19, 2020

karenzone May 19, 2020

sakthiskv commented May 20, 2020

sakthiskv commented May 20, 2020

thomhubers commented May 27, 2020

robbavey left a comment

pemontto commented May 27, 2020

thomhubers commented May 27, 2020 •

edited

robbavey commented May 27, 2020

karenzone left a comment •

edited

Alsatea commented Aug 10, 2020 •

edited

yuliansen commented Oct 8, 2020 •

edited

Feature attachments #48

Feature attachments #48

Conversation

pemontto commented Sep 25, 2019

pemontto commented Sep 26, 2019

pemontto commented Oct 18, 2019

robbavey left a comment

Choose a reason for hiding this comment

pemontto left a comment

Choose a reason for hiding this comment

karenzone left a comment

Choose a reason for hiding this comment

pemontto left a comment

Choose a reason for hiding this comment

sakthiskv commented May 19, 2020

pemontto commented May 19, 2020

sakthiskv commented May 19, 2020 • edited

pemontto commented May 19, 2020

sakthiskv commented May 19, 2020

pemontto commented May 19, 2020

pemontto commented May 19, 2020

karenzone left a comment

Choose a reason for hiding this comment

karenzone May 19, 2020

Choose a reason for hiding this comment

karenzone May 19, 2020

Choose a reason for hiding this comment

sakthiskv commented May 20, 2020

sakthiskv commented May 20, 2020

thomhubers commented May 27, 2020

robbavey left a comment

Choose a reason for hiding this comment

pemontto commented May 27, 2020

thomhubers commented May 27, 2020 • edited

robbavey commented May 27, 2020

karenzone left a comment • edited

Choose a reason for hiding this comment

Alsatea commented Aug 10, 2020 • edited

yuliansen commented Oct 8, 2020 • edited

sakthiskv commented May 19, 2020 •

edited

thomhubers commented May 27, 2020 •

edited

karenzone left a comment •

edited

Alsatea commented Aug 10, 2020 •

edited

yuliansen commented Oct 8, 2020 •

edited