Merge pull request #13 from wikiZ/main

Update Kunyu version v1.4.0

CHANGELOG

@@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [v1.4] - 2021-9-4
+### Added 
+- Modify the POC storage directory of the pocsuite module
+- Add HOST collision function
+- Example Change the output result set LAT and LON to Update time
+- Refactor part of the code
+
 ## [v1.3] - 2021-8-12
 ### Added 
 - Calculate hex/base64/mmh3/md5 	Command : [EncodeHash] 

README.md

@@ -31,7 +31,6 @@ For the use of kunyu, there can be many application scenarios, such as:
 
 ```
 git clone https://github.com/knownsec/Kunyu.git
-tar -xvf Kunyu.tar
 cd Kunyu
 pip3 install -r requirements.txt
 
@@ -42,9 +41,11 @@ Linux:
 Windows:
 	cd kunyu
 	python3 console.py
+
+PYPI:
+	pip3 install kunyu
 	
 P.S. Windows also supports python3 setup.py install.
-You can also use pip install kunyu to install and update.
 ```
 
 # 0x02 Configuration instructions
@@ -75,6 +76,7 @@ Global commands:
         SearchCert <Domain>                       SSL certificate Search
         SearchDomain <Domain>                     Domain name associated/subdomain search
         EncodeHash <encryption> <query>           Encryption method interface 
+        HostCrash <IP> <Domain>                   Host Header Scan hidden assets
         Seebug <Query>                            Search Seebug vulnerability information
         set <Option>                              Set arguments values
         Pocsuite3                                 Invoke the pocsuite component
@@ -126,6 +128,18 @@ Query through the serial number of the SSL certificate, so that the associated a
 
 ![](./images/searchcert.png)
 
+**Multi-factor query**
+
+Similarly, Kunyu also supports multi-factor conditional query related assets, which can be realized through ZoomEye logic operation syntax.
+
+![](./images/headersearch.png)
+
+**Feature Search**
+
+Through HTTP request packet features or website-related features, the same framework assets can be concatenated more accurately
+
+![](./images/factor.png)
+
 **Associated Domain/Subdomain Search**
 
 Search for associated domain names and subdomains, and query associated domain names by default. Two modes can be set by setting the dtype parameter.
@@ -160,6 +174,12 @@ In versions after v1.3.1, you can use kunyu to link the console mode of pocsuite
 
 ![](./images/pocsuite.png)
 
+**HOSTS head collision**
+
+Through the HOSTS collision, the hidden assets in the intranet can be effectively collided, and the intranet service can be accessed according to the ServerName domain name and IP configured in the middleware httpf.conf. This can be achieved by setting the local hosts file later, because the local hosts file takes precedence. The level is higher than DNS server resolution. Support reverse check through ZoomEye domain name library or read TXT file to get the list of domain names.
+
+![](./images/searchcrash.png)
+
 **Data result**
 
 All search results are saved in the user's root directory, and the directory is created based on the current timestamp. All query results of a single start are stored in an Excel format under one directory, giving a more intuitive experience. The output path can be returned through the ExportPath command.
@@ -221,7 +241,7 @@ Then save it and you can use it normally. The bug appears because there is a Chi
 
 **9. Pocsuite3 module POC storage directory**
 
-When using the pocsuite3 module, if you want to add a new POC module, you can add a POC file in **project directory/pocsuite3/pocs/**.
+When using the pocsuite3 module, if you want to add a new POC module, you can add a POC file in **project directory/kunyu/pocs/**.
 
 **10. Pocsuite3 module POC missing issue**
 

doc/README_CN.md

@@ -30,7 +30,6 @@ Kunyu(坤舆)，旨在让企业资产收集更高效，使更多安全相关从
 
 ```
 git clone https://github.com/knownsec/Kunyu.git
-tar -xvf Kunyu.tar
 cd Kunyu
 pip3 install -r requirements.txt
 
@@ -41,8 +40,11 @@ Linux:
 Windows:
 	cd kunyu
 	python3 console.py
+
+PYPI:
+	pip3 install kunyu
 	
-P.S. Windows同样支持python3 setup.py install，也可以使用pip install kunyu进行安装更新
+P.S. Windows同样支持python3 setup.py install.
 ```
 
 # 0x02 配置说明
@@ -73,6 +75,7 @@ Global commands:
         SearchCert <Domain>                       SSL certificate Search
         SearchDomain <Domain>                     Domain name associated/subdomain search
         EncodeHash <encryption> <query>     	  Encryption method interface 
+        HostCrash <IP> <Domain>                   Host Header Scan hidden assets
         Seebug <Query>                            Search Seebug vulnerability information
         set <Option>                              Set arguments values
         Pocsuite3                                 Invoke the pocsuite component
@@ -124,6 +127,18 @@ ZoomEye:
 
 ![](../images/searchcert.png)
 
+**特征搜索**
+
+通过HTTP请求包特征或网站相关特征可以进行更加精准的串并相同框架资产
+
+![](../images/headersearch.png)
+
+**多因素查询**
+
+同样kunyu也支持多因素条件查询关联资产，可以通过ZoomEye逻辑运算语法实现。
+
+![](../images/factor.png)
+
 **关联域名/子域名搜索**
 
 对关联域名以及子域名进行搜索，默认查询关联域名，可以通过设置 dtype 参数设置两种模式。
@@ -158,6 +173,12 @@ ZoomEye:
 
 ![](../images/pocsuite.png)
 
+**HOSTS头碰撞**
+
+通过HOSTS碰撞，可以有效的碰撞出内网中隐藏的资产，根据中间件httpf.conf中配置的ServerName域名和IP捆绑即可访问内网服务，后续通过设置本地hosts文件实现，因为本地hosts文件优先级高于DNS服务器解析。支持通过ZoomEye域名库反查或者读取TXT文件获取域名列表。
+
+![](../images/searchcrash.png)
+
 **数据结果**
 
 搜索的所有结果都保存在用户根目录下，并根据当前时间戳创建目录。单次启动的所有查询结果都在一个目录下，保存为Excel格式，给予更加直观的体验。可以通过ExportPath命令返回输出路径。

kunyu/config/__version__.py

@@ -15,7 +15,7 @@
 __python_version__ = sys.version.split()[0]
 __platform__ = platform.platform()
 __url__ = "https://github.com/knownsec/Kunyu"
-__version__ = '1.3.5'
+__version__ = '1.4.0'
 __author__ = '风起'
 __Team__ = 'KnownSec 404 Team'
 __author_email__ = 'onlyzaliks@gmail.com'
@@ -36,7 +36,7 @@
 
 GitHub:{url}
 
-kunyu is Cyberspace Resources Surveying and Mapping auxiliary tools
+kunyu is Cyberspace Search Engine auxiliary tools
 
 {{datil}}
 
@@ -50,7 +50,7 @@
 |_|\_\__,_|_| |_|\__, |\__,_| -v{version}
                  |___/         
 GitHub: https://github.com/knownsec/Kunyu/
-kunyu is Cyberspace Resources Surveying and Mapping auxiliary tools
+kunyu is Cyberspace Search Engine auxiliary tools
 {{datil}}
 """.format(version=__version__)
 

kunyu/config/setting.py

@@ -9,18 +9,32 @@
 '''
 
 import os
+import re
+
+# ZoomEye API
+USER_INFO_API = "https://api.zoomeye.org/resources-info"
+HOST_SEARCH_API = "https://api.zoomeye.org/host/search"
+WEB_SEARCH_API = "https://api.zoomeye.org/web/search"
+BOTH_SEARCH_API = "https://api.zoomeye.org/both/search"
+DOMAIN_SEARCH_API = "https://api.zoomeye.org/domain/search"
 
 # Setting ZoomEye Query Fields
-ZOOMEYE_FIELDS_HOST = ["ID", "IP", "Port", "Protocol", "Service", "ISP", "Country", "City", "Title", "Latitude", "Longitude"]
-ZOOMEYE_FIELDS_WEB = ["ID", "IP", "URL", "Title", "OS", "WebApp", "DB", "Language", "Server"]
+ZOOMEYE_FIELDS_HOST = ["ID", "IP", "Port", "Protocol", "Service", "ISP", "Country", "City", "Title", "Time"]
+ZOOMEYE_FIELDS_WEB = ["ID", "IP", "URL", "Title", "OS", "WebApp", "DB", "Language", "Server", "Time"]
 ZOOMEYE_FIELDS_INFO = ["Plan", "Search", "Stats", "Interval"]
 ZOOMEYE_FIELDS_DOMAIN = ["ID", "Domain", "IP", "TimeStamp"]
+HOST_SCAN_INFO = ["IP", "Domain", "Title"]
 
+# Kunyu OUTPUT File Path
 OUTPUT_PATH = os.path.expanduser('~/kunyu/output/')
 
 # REGEX rule
 IP_ADDRESS_REGEX = r"(([01]{0,1}\d{0,1}\d|2[0-4]\d|25[0-5])\.){3}([01]{0,1}\d{0,1}\d|2[0-4]\d|25[0-5])"
 HTTP_CHECK_REGEX = r"^https?:/{2}\w.+$"
+DOMAIN_CHECK_REGEX = r'^(([a-zA-Z]{1})|([a-zA-Z]{1}[a-zA-Z]{1})|'\
+    r'([a-zA-Z]{1}[0-9]{1})|([0-9]{1}[a-zA-Z]{1})|'\
+    r'([a-zA-Z0-9][-_.a-zA-Z0-9]{0,61}[a-zA-Z0-9])).'\
+    r'([a-zA-Z]{2,13}|[a-zA-Z0-9-]{2,30}.[a-zA-Z]{2,3})$'
 
 UA = [
         "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",

kunyu/core/__init__.py

@@ -13,6 +13,7 @@
 import json
 import argparse
 
+import grequests
 import requests
 import configparser
 

kunyu/core/crash.py

@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+# encoding: utf-8
+'''
+@author: 风起
+@contact: onlyzaliks@gmail.com
+@File: crash.py
+@Time: 2021/9/2 17:54
+'''
+import re
+import json
+import random
+
+import grequests
+import requests
+
+from kunyu.core import conf
+from rich.console import Console
+from kunyu.utils.log import logger
+from kunyu.utils.convert import convert
+from kunyu.lib.batchfile import get_domain_file
+from kunyu.config.setting import UA, DOMAIN_SEARCH_API, DOMAIN_CHECK_REGEX
+
+console = Console(color_system="auto", record=True)
+ZOOMEYE_KEY = conf.get("zoomeye", "apikey")
+ZOOMEYE_TOKEN = conf.get("login", "token")
+
+
+class HostScan:
+    def __init__(self):
+        self.headers = {
+            "User-Agent": random.choice(UA)
+        }
+        self.params = {"type": 1}
+        self.__get_login()
+
+    # Check whether the HTTP request returns an error
+    def __check_error(self, resp):
+        if resp.get("error"):
+            raise requests.HTTPError(resp.get("message"))
+
+    # Dynamically obtain user credentials according to the initialized configuration file
+    def __get_login(self):
+        if ZOOMEYE_KEY == "None":
+            self.headers["Authorization"] = "JWT %s" % ZOOMEYE_TOKEN
+        else:
+            self.headers["API-KEY"] = ZOOMEYE_KEY
+
+    # Get ZoomEye Doamin List
+    def __get_zoomeye_domain(self):
+        """
+            Return to the result set of the reverse search through the zoomeye domain name library
+            First get the number of result sets in the first HTTP request
+            Perform calculations based on the number of result sets to dynamically obtain all results
+        """
+        domain_list = []
+        try:
+            def request_get():
+                resp = requests.get(
+                    DOMAIN_SEARCH_API,
+                    data=self.params,
+                    headers=self.headers,
+                )
+                self.__check_error(json.loads(resp.text))
+                return resp.json()
+
+            result = request_get()
+            # Dynamically calculate the number of pages that need to be queried to obtain all result sets
+            count = int(result["total"] / 30)
+            page = count if (result["total"] % 30) == 0 else count + 1
+            console.log("Host Header Scan Domain Total: ", result["total"], style="green")
+            # Get ZoomEye Domain result
+            for i in range(page):
+                self.params["page"] = str(i + 1)
+                result = request_get()
+                for num in range(len(result["list"])):
+                    data = convert(result["list"][num])
+                    domain_list.append(data.name)
+
+            return domain_list
+
+        except requests.HTTPError as err:
+            logger.error(err)
+
+    def __is_valid_domain(self, search):
+        """
+            Return whether or not given value is a valid domain.
+            If the value is valid domain name this function returns ``True``, otherwise False
+            :param search: domain string to validate
+        """
+        pattern = re.compile(DOMAIN_CHECK_REGEX)
+        return True if pattern.match(search) else False
+
+    def _get_file(self, search):
+        """
+            Get the array of domain names required for HOST collision
+            If it is the main domain name, check it through the zoomeye interface
+            Otherwise, read the result through the file path
+            :param search: Enter the main domain name or domain name file path
+        """
+        domain_list = []
+        # Check the logic of obtaining the domain name result
+        if self.__is_valid_domain(search):
+            domain_list = self.__get_zoomeye_domain()
+        else:
+            for domain in get_domain_file(search):
+                domain_list.append(domain)
+            # Return to the domain name combing in the file
+            console.log("Host Header Scan Domain Total: ", len(domain_list), style="green")
+        return domain_list
+
+    def host_scan(self, search, ip):
+        """
+            Obtain hidden assets through HOST collision
+            :param search: Ways to obtain domain names
+            :param ip: IP address to be collided
+        """
+        crash_list = []
+        self.params["q"] = search
+        domain_list = self._get_file(search)
+        url = "http://{}/".format(ip)
+        resp = []
+        for domain in domain_list:
+            headers = {'Host': domain.strip(),
+                       'User-Agent': random.choice(UA)
+                       }
+            # Concurrent requests through the encapsulated coroutine module
+            resp.append(grequests.get(url, headers=headers, timeout=1))
+        res_list = grequests.map(resp)
+        for res in res_list:
+            try:
+                # Determine the HTTP status code that needs to be obtained
+                if res.status_code == 200 or res.status_code == 302 or res.status_code == 301:
+                    res.encoding = 'gbk2312'
+                    # Get the title of the returned result
+                    title = re.findall('<title>(.+)</title>', res.text)
+                    crash_list.append([ip, res.request.headers['Host'], title[0]])
+
+            except Exception:
+                continue
+
+        return crash_list