BOT: collector ftp(s) #455

robcza · 2016-02-28T17:10:38Z

Bots for collecting files from FTP and FTPS, both bots include automatic unzipping

Possible issue is no proxy support

codecov-io · 2016-02-28T17:16:32Z

Current coverage is `65.70%`

Merging #455 into master will decrease coverage by -0.74% as of c123202

@@            master   #455   diff @@
=====================================
  Files          175    179     +4
  Stmts         6750   6884   +134
  Branches         0      0       
  Methods          0      0       
=====================================
+ Hit           4485   4523    +38
  Partial          0      0       
- Missed        2265   2361    +96

Review entire Coverage Diff as of c123202

Powered by Codecov. Updated on successful CI builds.

sebix · 2016-02-28T17:19:40Z

Please write a least a dummy test, so the requirements and syntax will be checked as here: https://github.com/certtools/intelmq/blob/master/intelmq/tests/bots/collectors/alienvault_otx/test_collector.py

sebix · 2016-02-28T17:24:58Z

intelmq/bots/collectors/ftp/collector_ftps.py

+from intelmq.lib.message import Report
+
+
+# https://stackoverflow.com/questions/12164470/python-ftp-implicit-tls-connection-issue


If you copied code from there, attribution is required under cc by-sa 3.0: https://blog.stackoverflow.com/2009/06/attribution-required/

robcza · 2016-02-28T18:07:27Z

@sebix thank you for the comments, will fix and include the attribution

sebix · 2016-02-28T19:00:25Z

intelmq/bots/collectors/ftp/collector_ftp.py

+            report.add("feed.url", self.parameters.ftp_host + ':' +
+                       str(self.parameters.ftp_port), sanitize=True)
+            report.add("feed.accuracy", self.parameters.accuracy, sanitize=True)
+            time_observation = DateTime().generate_datetime_now()


this line is obsolete

sebix · 2016-03-05T12:41:44Z

intelmq/bots/collectors/ftp/collector_ftps.py

+from intelmq.lib.message import Report
+
+
+# BEGIN content from Stack Overflow


I'd also explicitly add cc by-sa 3.0

coveralls · 2016-03-20T21:52:51Z

Coverage decreased (-0.7%) to 65.703% when pulling 03e49f0 on robcza:ftp-collector into fac7721 on certtools:master.

sebix · 2016-03-29T10:11:49Z

Is it final?

aaronkaplan · 2016-05-11T21:51:27Z

ping? @robcza . Should we merge? Did you test it?
What about setups which need to go through a proxy?

robcza added 8 commits January 9, 2016 18:45

ftps (implicit) collector

2ca4870

logging improvements

d6d4685

int to str

fcef75d

smallfix

a806643

import zipfile

46256b3

BytesIO instead of StringIO

48a4827

BOT: added ftp collector

25a4e7a

ftp collector fixes

4ebc8e7

sebix self-assigned this Feb 28, 2016

sebix added the component: bots label Feb 28, 2016

sebix added this to the Release 1 - v1.0 milestone Feb 28, 2016

sebix reviewed Feb 28, 2016
View reviewed changes

robcza force-pushed the ftp-collector branch from ca869b1 to 92b93eb Compare February 28, 2016 18:30

sebix reviewed Feb 28, 2016
View reviewed changes

robcza force-pushed the ftp-collector branch from 92b93eb to e500997 Compare February 28, 2016 19:41

ftp(s) collector fixes, dummy tests and attribution

56acd5a

robcza force-pushed the ftp-collector branch from e500997 to 56acd5a Compare February 28, 2016 22:59

sebix reviewed Mar 5, 2016
View reviewed changes

sebix mentioned this pull request Mar 5, 2016

Automatic unzipping: problematic? #460

Closed

ENH: ftp(s) collector masked filenames and enhanced logging

c82e9d5

robcza force-pushed the ftp-collector branch from 61ac957 to c82e9d5 Compare March 20, 2016 21:00

ENH: ftp(s) collector default file masks

03e49f0

robcza force-pushed the ftp-collector branch from e942505 to 03e49f0 Compare March 20, 2016 21:47

sebix added the incomplete label Apr 30, 2016

sebix removed the incomplete label May 17, 2016

sebix merged commit 3c9958c into certtools:master May 17, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BOT: collector ftp(s) #455

BOT: collector ftp(s) #455

robcza commented Feb 28, 2016

codecov-io commented Feb 28, 2016

sebix commented Feb 28, 2016

sebix Feb 28, 2016

robcza commented Feb 28, 2016

sebix Feb 28, 2016

sebix Mar 5, 2016

coveralls commented Mar 20, 2016

sebix commented Mar 29, 2016

aaronkaplan commented May 11, 2016 •

edited

Loading

		from intelmq.lib.message import Report


		# https://stackoverflow.com/questions/12164470/python-ftp-implicit-tls-connection-issue

		from intelmq.lib.message import Report


		# BEGIN content from Stack Overflow

BOT: collector ftp(s) #455

BOT: collector ftp(s) #455

Conversation

robcza commented Feb 28, 2016

codecov-io commented Feb 28, 2016

Current coverage is 65.70%

sebix commented Feb 28, 2016

sebix Feb 28, 2016

Choose a reason for hiding this comment

robcza commented Feb 28, 2016

sebix Feb 28, 2016

Choose a reason for hiding this comment

sebix Mar 5, 2016

Choose a reason for hiding this comment

coveralls commented Mar 20, 2016

sebix commented Mar 29, 2016

aaronkaplan commented May 11, 2016 • edited Loading

Current coverage is `65.70%`

aaronkaplan commented May 11, 2016 •

edited

Loading