Google Summer of Code 2016 Work Product Submission

Evgeniya edited this page Aug 22, 2016 · 4 revisions

SNARE/TANNER

SNARE is a web application honeypot sensor attracting all sort of maliciousness from the Internet. The web page is generated by cloning a real web application and injecting known vulnerabilities. SNARE connects to TANNER, a remote data analysis and classification service, to evaluate HTTP requests and composing the response then served by SNARE.

TANNER

RFI Emulation

RFI (remote file inclusion) emulator extracts the remote file link from the requested path and downloads the file. All downloaded files are stored in /opt/tanner/scripts directory. Filename is created with md5 hash function. After downloading, the script body is sent to a PHP Sandox (https://github.com/mushorg/phpox), which is running on the TANNER host. PHP Sandbox executes this script and returns the result as a json that contains file hashsum and stdout of the execution.

https://github.com/mushorg/tanner/commit/4c4e991b1b39d8897dab4dbb71e4f37d306b161b https://github.com/mushorg/tanner/commit/83869ea80d0555c5e65e5bd4a17932304426c75d https://github.com/mushorg/tanner/commit/403ad61b2eff9117830da60983754970e4b28d9c https://github.com/mushorg/tanner/commit/4b613afb92eecc3e338fba398dd12f020ed07d76 https://github.com/mushorg/tanner/commit/afa018e49ad25bdc27acbeff52b41d4c928d0389 https://github.com/mushorg/tanner/commit/f2750987ee422ed5c5d8211ce681658d119d03eb

LFI Emulation

LFI (local file inclusion) emulation allows us to create a static virtual file system (virtualdocs folder), which contains certain system files (for example passwd, shadow, etc.) which are commonly of interest for an adversary. To create the virtualdocs structure, TANNER uses a definition in vdocs.json, where the structure and content of the files are described. On each LFI request, the LFI emulator gets the available files in virtualdocs folder. If the requested file is present, TANNER returns its content to SNARE.

https://github.com/mushorg/tanner/commit/392c6250ccf723b4443aabb3250baeb169078d37 https://github.com/mushorg/tanner/commit/1399b18753fd9ac764745128560cf959c6aac1ba https://github.com/mushorg/tanner/commit/93a759f9e14403bff528f8939c4eff5d262a7faf

XSS Emulation

XSS (Cross-site scripting) data can come from HTTP POST and GET requests.

  • If the XSS injected data comes from a POST request, we get the data from the request body.
  • If the XSS injected data comes from a GET request, we extract the data from the request parameters.

After that, we choose a page into which we will inject the script. For this purpose we get the session paths and get the last path with type ‘text/html’. The script is injected in SNARE’s response body.

https://github.com/mushorg/tanner/commit/27af9dae3c90e62a03a8c01a6de2a95591fab04d https://github.com/mushorg/tanner/commit/c98b0cf9dc78e1d1ed5f3f3dac59c1ced2cbb94a https://github.com/mushorg/tanner/commit/64dad53bc367afc36954e5f2f02b67cb55eed445 https://github.com/mushorg/tanner/commit/2849ae66d97959d1390b91a2363b55e887ecbcdd https://github.com/mushorg/snare/commit/9c45618fcdc5280700620e15da36890ebeae01c8

Sessions and session tracking

This module creates sessions with information coming from SNARE and tracks session paths. Info from SNARE is validated before used.

Every session contains:

  • Peer ip and port
  • Peer user agent
  • UUID of the SNARE sensor
  • Requested paths: path, timestamp and SNARE response status
  • Session UUID
  • Start timestamp
  • Current Timestamp
  • Count of requests

Uniqueness of the session is determined by the session ip and user agent. Every session has an expiration time, by default this time equals 75s. After expiration, the session is deleted from an active session and analyzed.

https://github.com/mushorg/tanner/commit/6fa56408fb1bc7d7397b5814ee09f9971317a860 https://github.com/mushorg/tanner/commit/110a1d20a03d9e6f92b1aa68274aa25a797ce0f0 https://github.com/mushorg/tanner/commit/99747b2a61a4dfae7707260c2aa3b049045644de https://github.com/mushorg/snare/commit/1e15e680643076dcab462eece53633d6a119c434 https://github.com/mushorg/snare/commit/3fd8b5f91eba809db1c409d024113073b298082e

Session evaluating

Evaluate session info after the session becomes inactive. Session evaluation mechanism analyzes session info and can suggest possible owners. The result of the analysis contains:

  • Session attributes
    • UUID
    • Peer ip and port
    • User agent
    • Sensor_uuid
  • Start time
  • End time - last session timestamp
  • Requests per second
  • Approximate time between requests
  • Accepted paths - number of accepted paths
  • Errors - counts of errors in SNARE responses
  • Hidden links - count of accepted dorks
  • Attack types - list of attack types
  • Paths - list of all paths
  • Possible owners - list of possible owners. May be user, attacker, tool and crawler

https://github.com/mushorg/tanner/commit/7087bb8db8bca72614e734abdb4f9ce547e17f94 https://github.com/mushorg/tanner/commit/432deacb200779bacaacfd2015e6ff4dd22e8a22 https://github.com/mushorg/tanner/commit/03315b4256963290e1ea3d25b275fb81681b4bdb

Dorks Managing

Managing manually added dorks and extract user dorks from requested paths. Extracting dorks from the requested paths allows us to make SNARE more attractive for an attacker by adding unknown vulnerable paths and permanently expand the dork database. When SNARE requests the dorks, TANNER returns a random number of manually added dorks and extracted dorks. https://github.com/mushorg/tanner/commit/ebf2f6afa5b2825323c04fafe8970aafe542d8ec

SQL Injection emulation

Implementation of the SQLi emulator was the most tricky part of my work. There are a lot of improvements that can be done in the future, but basic work was done.

SQLi emulation process:

  • Setup main db
  • Detect sqli with libinjection
  • Create attacker’s db
  • Map the sqli requests to the db table
  • Execute query on attacker’s db
  • Return the result

https://github.com/mushorg/tanner/commit/dde3110c17330da568cab01e155febde62780ecc https://github.com/mushorg/tanner/commit/d499b1b76b8a768c03bc919f889800796039c070 https://github.com/mushorg/tanner/commit/8e14590df76627da33e943c10e1ddb30652f4965 https://github.com/mushorg/tanner/commit/4e8e98a6d37e4a44633d4c32cbe7edaa4b4ba945 https://github.com/mushorg/tanner/commit/e94a74c2b0ffce18ef83fca8c9f25498cb8b2cb0 https://github.com/mushorg/tanner/commit/ddac1f85464a743b574fd7e65d4c31cb99ec80df https://github.com/mushorg/tanner/commit/0647d8072b0aed8736e7cfba6d757cfbb72433de

Basic Api.

Basic api allows get information about snare sensors, which communicate with TANNER and get sessions info from certain SNARE sensor.

https://github.com/mushorg/tanner/commit/38925403067249cf9e526672e0eb3f26ae27c84d

Optimization

https://github.com/mushorg/tanner/commit/5f076665eabf1101a8dbf1bee1789e1c096d7c04 https://github.com/mushorg/tanner/commit/b418d4497c6cad9f019392463b4e49b8d9496a88

Logging

We add logging system instead of print messages. https://github.com/mushorg/tanner/commit/baadc53e38c48b3f77ab07e1b000f4f06d9459b2

SNARE

Cloning process

Cloning process was improved. Before that only the main page was cloned correctly. Now the site is cloned recursively. During a cloning process all links are being replaced. It allows to navigate across the cloned site.

https://github.com/mushorg/snare/commit/ba4c64b3cec907960a695b69e78e025a7e77c083 https://github.com/mushorg/snare/commit/7bb381eb0f6a84abb466c280755c070e550ac1b2 https://github.com/mushorg/snare/commit/f14a63cbfd0c350d7e084ccfaeaa55574c760d71

Webmaster tools

SNARE supports Google and Bing webmaster tools. https://github.com/mushorg/snare/commit/0b944137f50d00ab0793e6032249e580c531de83

Auto-update

Auto-update system allows to update SNARE without stopping it. User can configure update timeout. Possible timeouts is: D - day, H - hours, M- minutes. SNARE use gitpython for checking updates. If updates are available, gitpython downloads the updates and SNARE restarts. https://github.com/mushorg/snare/commit/99a8c4fa1f38edc922516a467cfac8d918184023

Misc

https://github.com/mushorg/snare/commit/8136a52e0bd6dec802f1ed7f00bacce16cf8e295 https://github.com/mushorg/tanner/commit/7ec265551889475ff62aa10f1cd28a6af8103b64 https://github.com/mushorg/tanner/commit/638ed6aab608e5fce8bf80e13329518996e9d31f https://github.com/mushorg/tanner/commit/32495fb774af72981a53feb20e272af124bc65b8 https://github.com/mushorg/tanner/commit/56265a3100b7f201c0c0a0f10e65ca450094e9fc https://github.com/mushorg/tanner/commit/f6f966e9ced32dec93f4ed82de0bc952bc0e248a

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.