Sending out x-arf emails

[bernhardreiter]

Should be able to send out x-arf emails.
Specification available from http://www.x-arf.org,
the question is: which version v0.2 or v0.3 draft.

TODO: List x-arf sender and receivers. Look for example emails.

[bernhardreiter]

Some examples can be found from http://www.x-arf.org/schemata.html

Overall the x-arf community seems to have stalled in the last 2-3 years, indicators:

• Website http://www.x-arf.org/tools.html does not link https://pypi.python.org/pypi/pyxarf
• https://github.com/abusix/xarf-schemata mentioned expired JSON-schema draft 02: http://tools.ietf.org/html/draft-zyp-json-schema-02
• Last format spec commit in Dec 2013 https://github.com/abusix/xarf-specification/commits/master
• https://docs.abusix.com/pyxarf/ home page of the python parser is missing.
• The official mailinglist debate has only one mail from Oct 2014 an unanswered question.

[bernhardreiter]

Analysing the two existing python implementation I've found:

http://x-arfreporting.sourceforge.net/

• Using the xarf.zip file which is newer than the code repo files, all from 2010.
• python3.2.3 barfs on xarg.py
  ( File "xarf.py", line 61, in generateMessageHeader
  self.Message = email.mime.Multipart.MIMEMultipart()
  AttributeError: 'module' object has no attribute 'mime')
  so it is not python3 ready.
  Uncommenting out the dnspython and setting and explicit abuse address and
  a local mailserver, the test python2 xarf.py runs.
• License: Gnu GPL v>=3: compatible with GNU AGPL
• Only sets X-ARF: YES header which is v0.1 x-arf.
  Does not check the schema, only uses
  http://www.x-arf.org/schema/abuse_malware-attack_0.1.0.json
  http://www.x-arf.org/schema/abuse_login-attack_0.1.0.json
• Its small
  wc -l *.py
  54 authlogSSHParser.py
  52 nepenthesParser.py
  87 reportit.py
  163 xarf.py
  356 total
• uses dnspython to ask abusix for the contact address.
• xarf.py generates the emails and basically only fills the a-arf v0.1 structure.
  Email generation is not needed for intelmq-mailgen.

=> not helping much, could more easily be reimplemented in python3

https://pypi.python.org/pypi/pyxarf

• Using the https://bitbucket.org/abusix/pyxarf.git version which has more files than
  pyxarf-src-0.0.5.tar.gz
• python2 only.
• License: Apache 2.0 compatible with GNU AGPL.
• Uses dnspython, ipaddr and querycontacts only in xarfmail/xarfmail.py where mail is send out=> both not needed for intelmq
• xarfmail.py sets X-XARF = PLAIN, so it is v0.2 x-arf
• scripts/xarfutil.py can use command lines to set all values, => also not needed
• xarf.py validates schemas, downloads and caches them. Has tricks to convert validation schema
  to draft3 of the json specs to be able to validate them. => interesting, more for reading, than for sending.
• small, sloccount has
  SLOC Directory SLOC-by-Language (Sorted)
  404 scripts python=404
  371 pyxarf python=371
  112 xarfmail python=112
  57 top_dir python=57
  51 tests python=40,sh=11
  20 docs python=20

python: 1004 (98.92%)
sh: 11 (1.08%)

=> useful as basis for a base library that could parse, validate and write xarf.
Would need removal of unwanted parts, port to python3.

[bernhardreiter]

Further design thoughts:

• It would be interesting to know how many xarf receivers can and would like to have XARF:BULK.
  In absense of this knowledge we could start with implement the single email version first
  and then have two format variants later, if needed.
• If pyxarf is a well used library if would make sense to maintain it in compatibility with python2 and python3. => Should try to contact them.
• the interface from intelmq-mailgen to any xarf library would include parsing, validating, writing of the yaml part, creating the email structure and giving back a list of email structure which could then be filled with additional information. Because each BULK email would need a full header, there must be a way to give this to the library before construction as well.
• given that pyxarf is small, for intelmq there is the extra burden of maintaining python2 compatibility, maintaining it as a library, maintaining functionality that is not necessary for intelmq (like email sending, command line interface) It may make more sense to just create an xarf functionality within intelmq-mailgen with the reduced functionality. That is an open question.

[bernhardreiter]

https://bitbucket.org/abusix/pyxarf.git did not enable tickets so I've send an email to info@abusix.com
(as official contact for pyxarf) asking if this is still active and where a public place to ask questions is.

[bernhardreiter]

Talked to @dmth : We will try the "integrate what we need directly in intelmq first" approach.

Rationale: We find out faster, what we actually need in terms of interface and python3 functions. And we provide first value faster. Later we can see how active the pyxarf community is to extend pyxarf to that use case. We also know our technical requirements better at that point, having it tried out. If pyxarfcan we developed in that direction, we can refactor to use the updated library later.

[bernhardreiter]

pyxarf does not need a lot of changes to run on python3.
Thus I've created a fork https://bitbucket.org/bereiter/pyxarf which I assume will be installed
(currently from the master branch) for intelmq-mailgen xarf branch I am working.

[bernhardreiter]

Current status in branch xarf is that you can see the example mapping for malware and multiple ones get written out into the text body part for debugging purposes. Mail creation needs to be refactored.

[bernhardreiter]

#7 of branch xarf provides basic xarf writing support
with experimental malware mapping. The mapping need to be improved with close-to-real test data.

[bernhardreiter]

The basic xarf support from the branch has been merged to master,
so that the structural changes done can be used for other formats as well.

[bernhardreiter]

Meanwhile the rudimentary experimental xarf sending support was removed 23b07eb
and a new branch https://github.com/Intevation/intelmq-mailgen/tree/x-arf-mapping
provides first attempts at a semantic mapping.

My next step: Finding example data testing the mapping.

[bernhardreiter]

An small update:
@dmth is currently working on x-arf support, especially on the algorithms for mapping values.
We've created an example mapping for shadowserver-botnet-drone data to start somehwere
and engaged with the x-arf community about how to deal with missing values, see
https://github.com/Intevation/intelmq-mailgen/wiki/X-ARF

This issue is more specifically about implementing writing with intelmq-mailgen, which has a dependency on the currently by @bernhard-herzog developed new way of configuring how events are distributed.

[bernhardreiter]

@dmth what is the status of this?

[dmth]

https://github.com/Intevation/intelmq-mailgen/blob/master/example_scripts/20xarf.py contains an implementation of the newly developed schema. But it can only send X-ARF "simple", "bulk" is not supported yet. The Script is user-maintainable and can be extended with own or official schemata.

• We achieved an implementation of X-ARF simple into mailgen
• I recommend to close this issue and create new issues for extensions and enhancements if necessary.