数据采集、处理、可视化

工具箱

gocolly/colly: Elegant Scraper and Crawler Framework for Golang https://github.com/gocolly/colly

Scraping Framework for Golang http://go-colly.org/

Echo - High performance, minimalist Go web framework https://echo.labstack.com/

labstack/echo: High performance, minimalist Go web framework https://github.com/labstack/echo

MongoDB for GIANT Ideas | MongoDB https://www.mongodb.com/

Node.js https://nodejs.org/en/

加密与解密

20180903

站点加密结果：

{"r":"ZXlKM1pXSlRhWFJsSWpvaUlpd2ljV2xoYjAxaGNDSTZleUpqYjJSbElqb2lNVEV4TVNKOUxDSndZMDVoZGtsdVptOUVkRzlNYVhOMElqcGJleUpsYms1aGJXVWlPaUpvYjIxbElpd2libUYyVG1GdFpTSTZJdW1tbHVtaHRTSXNJbTVoZGxWeWJDSTZJbWgwZEhBNkx5OTNkM2N1WTI1b1lXNHVZMjl0TDNOb1lXNTBkV2t2YUc5dFpTMHpOamt4TG5Ob2RHMXNJbjBzZXlKbGJrNWhiV1VpT2lKaFltOTFkQ0lzSW01aGRrNWhiV1VpT2lMbGhhemxqN2pucm9Ea3U0c2lMQ0p1WVhaVmNtd2lPaUpvZEhSd09pOHZkM2QzTG1OdWFHRnVMbU52YlM5emFHRnVkSFZwTDJGaWIzVjBMVE0yT1RFdWMyaDBiV3dpZlN4N0ltVnVUbUZ0WlNJNkluQnliMlIxWTNScGIyNGlMQ0p1WVhaT1lXMWxJam9pNUxxbjVaT0I1YkdWNTZTNklpd2libUYyVlhKc0lqb2lhSFIwY0RvdkwzZDNkeTVqYm1oaGJpNWpiMjB2YzJoaGJuUjFhUzl3Y205a2RXTjBhVzl1TFRNMk9URXVjMmgwYld3aWZTeDdJbVZ1VG1GdFpTSTZJbUYxZEdnaUxDSnVZWFpPWVcxbElqb2k1THlCNUxpYTZMV0U2TFNvSWl3aWJtRjJWWEpzSWpvaWFIUjBjRG92TDNkM2R5NWpibWhoYmk1amIyMHZjMmhoYm5SMWFTOWhkWFJvTFRNMk9URXVjMmgwYld3aWZTeDdJbVZ1VG1GdFpTSTZJbU5oYzJVaUxDSnVZWFpPWVcxbElqb2k1cUdJNUw2TDVMaXQ1YitESWl3aWJtRjJWWEpzSWpvaWFIUjBjRG92TDNkM2R5NWpibWhoYmk1amIyMHZjMmhoYm5SMWFTOWpZWE5sTFRNMk9URXVjMmgwYld3aWZTeDdJbVZ1VG1GdFpTSTZJbUZ5ZEdsamJHVWlMQ0p1WVhaT1lXMWxJam9pNXBhdzZaZTc1TGl0NWIrRElpd2libUYyVlhKc0lqb2lhSFIwY0RvdkwzZDNkeTVqYm1oaGJpNWpiMjB2YzJoaGJuUjFhUzloY25ScFkyeGxMVE0yT1RFdWMyaDBiV3dpZlN4N0ltVnVUbUZ0WlNJNkltTnZiWEJoYm5raUxDSnVZWFpPWVcxbElqb2k1THlCNUxpYTZhT082WWVISWl3aWJtRjJWWEpzSWpvaWFIUjBjRG92TDNkM2R5NWpibWhoYmk1amIyMHZjMmhoYm5SMWFTOWpiMjF3WVc1NUxUTTJPVEV1YzJoMGJXd2lmU3g3SW1WdVRtRnRaU0k2SW1OdmJuUmhZM1FpTENKdVlYWk9ZVzFsSWpvaTZJR1U1N083NW9pUjVMdXNJaXdpYm1GMlZYSnNJam9pYUhSMGNEb3ZMM2QzZHk1amJtaGhiaTVqYjIwdmMyaGhiblIxYVM5amIyNTBZV04wTFRNMk9URXVjMmgwYld3aWZTeDdJbTVoZGs1aGJXVWlPaUxvdjV2bGhhWGxycGpudlpFaUxDSnVZWFpWY213aU9pSm9kSFJ3T2k4dkluMWRMQ0p0YjJKcGJHVk9ZWFpKYm1adlJIUnZUR2x6ZENJNlczc2libUYyVG1GdFpTSTZJdW1tbHVtaHRTSXNJbTVoZGxWeWJDSTZJbWgwZEhBNkx5OTNkM2N1WTI1b1lXNHVZMjl0TDNOb1lXNTBkV2t2YUc5dFpTMHpOamt4TG5Ob2RHMXNJbjBzZXlKdVlYWk9ZVzFsSWpvaTVMcW41Wk9CNWJHVjU2UzZJaXdpYm1GMlZYSnNJam9pYUhSMGNEb3ZMM2QzZHk1amJtaGhiaTVqYjIwdmMyaGhiblIxYVM5d2NtOWtkV04wYVc5dUxUTTJPVEV1YzJoMGJXd2lmU3g3SW01aGRrNWhiV1VpT2lMa3ZJSGt1SnJvdFlUb3RLZ2lMQ0p1WVhaVmNtd2lPaUpvZEhSd09pOHZkM2QzTG1OdWFHRnVMbU52YlM5emFHRnVkSFZwTDJGMWRHZ3RNelk1TVM1emFIUnRiQ0o5TEhzaWJtRjJUbUZ0WlNJNkl1YVdzT21YdStTNHJlVy9neUlzSW01aGRsVnliQ0k2SW1oMGRIQTZMeTkzZDNjdVkyNW9ZVzR1WTI5dEwzTm9ZVzUwZFdrdllYSjBhV05zWlMwek5qa3hMbk5vZEcxc0luMHNleUp1WVhaT1lXMWxJam9pNklHVTU3Tzc1b2lSNUx1c0lpd2libUYyVlhKc0lqb2lhSFIwY0RvdkwzZDNkeTVqYm1oaGJpNWpiMjB2YzJoaGJuUjFhUzlqYjI1MFlXTjBMVE0yT1RFdWMyaDBiV3dpZlYwc0luQnliMlIxWTNSSmJtWnZSSFJ2VEdsemRDSTZXMTBzSW1OdmJuUmhZM1JKYm1adlJIUnZJanA3SW1Ga1pISmxjM01pT2lMbm01RGxzYkhsanIvb2tyTG10THpsdklEbGo1SGxqTG9pTENKamIyMXdUbUZ0WlNJNkl1YXlwK1czbnVXUGkrbUFtdWV1b2VtQmsrYWNpZW1aa09XRnJPV1B1Q0lzSW1OdmJuUmhZM1JPWVcxbElqb2k1YTJaNTd1UDU1Q0dJaXdpWlcxaGFXd2lPaUl4TnpJM056azVNREJBY1hFdVkyOXRJaXdpWm1GNElqb2lNRE14TnpZd09EZzFOemNpTENKb2IzUk1hVzVsSWpvaU1UZzJNVGMzTWpneE5UZ2lMQ0pzWldaMFFXUkpiV2RWY213aU9pSm9kSFJ3T2k4dmVYVnVkSFZyZFM1emFDMXpaVzh1WTI5dEwyZHliM1Z3TVM5Tk1EQXZNREl2UmprdmNrSk5TMHRHZFVOTGFFOXdNVGR5WDBGQlFrUTFSM1JUVFhVNE5UTXdMbXB3WnlJc0lteGxablJCWkU1aGJXVWlPaUxrdUpQa3VKcm9ucnJtbDR2cGtxTG5ycUhubEova3VxY2lMQ0pzWldaMFFXUlZjbXdpT2lJaUxDSnNiMmR2VlhKc0lqb2lhSFIwY0RvdkwzbDFiblIxYTNVdWMyZ3RjMlZ2TG1OdmJTOW5jbTkxY0RFdlRUQXdMekF3TDBSRkwzSkNUVXRMUm5KNFVESTJNRWs0WVd4QlFVRmFNVVEwUzE5T1NUVXlOaTVxY0djaUxDSnRiMkpwYkdWTWIyZHZWWEpzSWpvaWFIUjBjRG92TDNsMWJuUjFhM1V1YzJndGMyVnZMbU52YlM5bmNtOTFjREV2VFRBd0x6QXlMMFk1TDNKQ1RVdExSblZEUzJsSFJUVkJjMGhCUVVGR2FGbExhRmROTkRNeU1pNXFjR2NpTENKdGIySnBiR1ZRYUc5dVpTSTZJakU0TmpFM056STRNVFU0SWl3aWNISnZlSGxWYzJWeVNXUWlPakl5TXl3aWNYRXhJam9pTVRjeU56YzVPVEF3SWl3aWNYRXlJam9pSWl3aWMyVnlkbVZ5UTI5dGNFNWhiV1VpT2lMbXNyUGxqSmZtZzUvbW1JN252WkhudTV3aUxDSnpaWEoyWlhKRGIyMXdWR1ZzSWpvaU1UZ3hNelF6TnpnNE9EWWlMQ0p6WlhKMlpYSkRiMjF3VjJWaWMybDBaU0k2SW1oMGRIQTZMeTkzZDNjdWFHSjNiWGg0TG1OdmJTSXNJblJsYkNJNklqRTROakUzTnpJNE1UVTRJaXdpZEdWc1VHaHZibVVpT2lJeE9EWXhOemN5T0RFMU9DSXNJbmRoYm1kM1lXNW5Jam9pSWl3aWQyVmlVMmwwWlNJNklpSjlMQ0p0YjJKcGJHVlhaV0pUYVhSbElqb2lJaXdpWTJGeWIzVnpaV3hKYm1adlJIUnZUR2x6ZENJNlczc2lhVzFuVlhKc0lqb2lhSFIwY0RvdkwzbDFiblIxYTNVdWMyZ3RjMlZ2TG1OdmJTOW5jbTkxY0RFdlRUQXdMekF5TDBZNUwzSkNUVXRMUm5WRFRFb3ljR3RKTmtaQlFVbDVaakUxUzA5TFl6TTFNaTVxY0djaUxDSnRiMkpwYkdWSmJXZFZjbXdpT2lKb2RIUndPaTh2ZVhWdWRIVnJkUzV6YUMxelpXOHVZMjl0TDJkeWIzVndNUzlOTURBdk1ESXZSamt2Y2tKTlMwdEdkVU5NVDFNeVFsbDROa0ZCUW5wbE1HZHRlVzF2TkRNd0xtcHdaeUo5TEhzaWFXMW5WWEpzSWpvaWFIUjBjRG92TDNsMWJuUjFhM1V1YzJndGMyVnZMbU52YlM5bmNtOTFjREV2VFRBd0x6QXpMekE1TDNKQ1RVdExSblZFZGpoeWVVbHFNemRCUVVsak5VNXlXbXhrYnpnd015NXFjR2NpTENKdGIySnBiR1ZKYldkVmNtd2lPaUpvZEhSd09pOHZlWFZ1ZEhWcmRTNXphQzF6Wlc4dVkyOXRMMmR5YjNWd01TOU5NREF2TURJdlJqa3Zja0pOUzB0R2RVTk1VR2x0VmxSVldFRkJRalpxZWtoa1dFMXZNVE0zTG1wd1p5SjlMSHNpYVcxblZYSnNJam9pYUhSMGNEb3ZMM2wxYm5SMWEzVXVjMmd0YzJWdkxtTnZiUzluY205MWNERXZUVEF3THpBekx6QkJMM0pDVFV0TFJuVkVkMHBEVEhjM1MxcEJRVVZzUVRkRlVXdDZZems1TkM1cWNHY2lMQ0p0YjJKcGJHVkpiV2RWY213aU9pSm9kSFJ3T2k4dmVYVnVkSFZyZFM1emFDMXpaVzh1WTI5dEwyZHliM1Z3TVM5Tk1EQXZNREl2UmprdmNrSk5TMHRHZFVOTVVYVkJjbkpyTUVGQlFraENRalV3ZUc1ak5UQTFMbXB3WnlKOUxIc2lhVzFuVlhKc0lqb2lhSFIwY0RvdkwzbDFiblIxYTNVdWMyZ3RjMlZ2TG1OdmJTOW5jbTkxY0RFdlRUQXdMekF6THpCQkwzSkNUVXRMUm5WRWQwdFBMVU55V1VOQlFVZFNkMmt3ZW1WdVVUa3dPUzVxY0djaUxDSnRiMkpwYkdWSmJXZFZjbXdpT2lKb2RIUndPaTh2ZVhWdWRIVnJkUzV6YUMxelpXOHVZMjl0TDJkeWIzVndNUzlOTURBdk1ESXZSamt2Y2tKTlMwdEdkVU5NVTBkaWJ6UmZNVUZCUWxrNWFVaG9URE5WT0RFNExtcHdaeUo5TEhzaWFXMW5WWEpzSWpvaWFIUjBjRG92TDNsMWJuUjFhM1V1YzJndGMyVnZMbU52YlM5bmNtOTFjREV2VFRBd0x6QXlMMFk1TDNKQ1RVdExSblZEVEVGWExWTnBZVkZCUVVkNU0yVTNMVjgzVlRnNE1DNXFjR2NpTENKdGIySnBiR1ZKYldkVmNtd2lPaUpvZEhSd09pOHZlWFZ1ZEhWcmRTNXphQzF6Wlc4dVkyOXRMMmR5YjNWd01TOU5NREF2TURJdlJqa3Zja0pOUzB0R2RVTk1RVmN0VTJsaFVVRkJSM2t6WlRjdFh6ZFZPRGd3TG1wd1p5SjlYWDA9"}

站点解密过程：

http://www.cnhan.com/shantui//templates/MC530/TP001/js/template.js

function decode(data) {
    var a = CryptoJS.enc.Base64.parse(data);
    return a.toString(CryptoJS.enc.Utf8);
}
var qiaoContent="0";//0默认没有商桥内容 1有商桥
$.ajax({
    url: window.platformPath + '/dynamic/get/data/allCompanyInfoByCompCode.json?compCode=' + window.cCode,
    dataType: 'json',
    success: function(data){
        var str = decode(data.r);
        var str2 = decode(str);
        str2 = JSON.parse(str2)
        var contactInfoDto = str2.contactInfoDto;
        var pcNavInfoDtoList = str2.pcNavInfoDtoList;
        var productInfoDtoList = str2.productInfoDtoList;
        var carouselInfoDtoList = str2.carouselInfoDtoList;
        var mobileNavInfoDtoList = str2.mobileNavInfoDtoList;
        var qiaoMap = str2.qiaoMap;
        // 1.
        MC500TP001site_footer(contactInfoDto);
        MC500TP001contact_us(contactInfoDto);
        MC500TP001company_card(contactInfoDto);
        MC500TP001mobile_footer(contactInfoDto);
        // 2.
        MC500TP001site_header(contactInfoDto,pcNavInfoDtoList);
        // 3.
        MC500TP001businesses(productInfoDtoList);
        // 4.
        MC500TP001carousel(carouselInfoDtoList);
        // 5.
        MC500TP001mobile_nav(mobileNavInfoDtoList);
        // 6.
        MC500TP001Qiao(qiaoMap);
    },
    error: function(data){
        console.log(data);
    }
});

代码解密方案：

借助nodejs解析加密字符串，node安装库较python方便。

20180911

http://cn.sonhoo.com/wukong/a58627

至今持续约1周，换浏览器、物理机，都是403。

<title>403 Forbidden</title>

403 Forbidden

nginx/1.13.12

CF-RAY: 4588cc1f12c34d02-CSX
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html
Date: Tue, 11 Sep 2018 08:34:48 GMT
P3P: CP=CAO PSA OUR
Server: yunjiasu-nginx
Transfer-Encoding: chunked
X-Powered-By: ARR/3.0
X-Powered-By: ASP.NET

代码解决方案：

	rp, err := proxy.RoundRobinProxySwitcher("http://121.69.37.6:9797", "http://112.95.18.143:8088")
	if err != nil {
		log.Fatal(err)
	}
	c.SetProxyFunc(rp)