Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'] #86

Closed
wushifeng opened this issue Jan 12, 2022 · 8 comments
Closed

NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'] #86

wushifeng opened this issue Jan 12, 2022 · 8 comments

Comments

@wushifeng
Copy link

由于是windows机器，因此使用和server一样的php镜像做了个客户端的镜像，如下，将一些github下载慢的包等手动安装了【不手动安装也一样有问题】：

FROM python:3.7.12-slim

#肯能用不了这么多的软件包，为了快速构建使用server的构建缓存，保留一致
ARG EXTRA_TOOLS="gnupg curl wget jq vim-tiny net-tools procps python3-dev default-libmysqlclient-dev locales inotify-tools gcc subversion git telnet iputils-ping vim openssh-client"
COPY ./sources.list /etc/apt/sources.list

RUN set -ex && cd / \
    && apt-get update \
    && apt-get install -y --no-install-recommends $EXTRA_TOOLS \
    && apt-get update \
    && apt-get install -y mariadb-client \
    && apt-get clean \
    && echo "LC_ALL=zh_CN.UTF-8" >> /etc/environment \
    && echo "zh_CN.UTF-8 UTF-8" > /etc/locale.gen \
    && echo "LANG=zh_CN.UTF-8" > /etc/locale.conf \
    && locale-gen \
    && ln -sf /usr/share/zoneinfo/Asia/Hong_Kong /etc/localtime \
    && rm -rf /var/cache/apt/* /root/.cache

COPY ./client /opt/client

RUN apt-get install -y flex bison #phantomjs
# https://phantomjs.org/download.html 下载 phantomjs
# docker cp .\pkg\phantomjs tca-client:/usr/bin/
COPY ./pkg/phantomjs /usr/bin/phantomjs
RUN apt-get install -y bzip2 fontconfig && \
    chmod +x /usr/bin/phantomjs && \
    echo "export OPENSSL_CONF=/etc/ssl/" >> /etc/profile

RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r /opt/client/requirements/app_reqs.pip
#cobra
RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r /opt/client/requirements/tool/cobra_requirements.txt

#从https://github.com/TCATools/下载的个工具打包后执行展开放入
ADD ./pkg/datatool.tar.gz /opt/client/

# https://hub.fastgit.org/Tencent/CodeAnalysis/issues/42
#RUN /opt/client/requirements/install.sh && \
RUN pip install pylint semgrep && \
    mkdir -p /opt/code
RUN ln -s /usr/lib/x86_64-linux-gnu/libffi.so.7 /usr/lib/x86_64-linux-gnu/libffi.so

然后，运行这个客户端 docker run -it --name tca-client --network host tca-client:2022 sh
执行 cd /opt/client/; python3 codepuppy.py localscan 出现如下的错误

[Tools init]: 100%|█████████████████████████████████████████████████| 13/13 [11:13<00:00, 51.81s/it]
2022-01-12 15:08:55,985-INFO-util.logutil: Initialize tools done.(use time: 673.59s)
2022-01-12 15:08:57,514-ERROR-util.logutil: cobra不可用，建议客户使用data/tools/cobra-v2.0.0-alpha.5/install.sh脚本进行
安装！
2022-01-12 15:08:57,515-INFO-util.logutil:
****************************************************************************************************
代码分析结果:
NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'].
****************************************************************************************************
2022-01-12 15:08:57,515-INFO-util.logutil: json report: /opt/client/scan_status.json
2022-01-12 15:08:57,515-ERROR-util.logutil: NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'].
Traceback (most recent call last):
  File "/opt/client/node/localrunner.py", line 380, in run
    request_generator.generate_request(proj_conf)
  File "/opt/client/node/localtask/requestgenerator.py", line 153, in generate_request
    execute_request_list = self._get_execute_request_list(job_context, task_list)
  File "/opt/client/node/localtask/requestgenerator.py", line 183, in _get_execute_request_list
    raise NodeError(code=errcode.E_NODE_TASK_CONFIG, msg=f"当前环境不支持{task_name}工具步骤:{remote_procs}.")
util.exceptions.NodeError: Error[204]: 当前环境不支持cobra工具步骤:['analyze'].

按照 #42 问题的描述，这个是完全正常的
# /opt/client/data/tools/common-cobra-v2.0.0-alpha.5/cobra.py --help
usage: cobra [-h] [-t <target>] [-f <format>] [-o <output>] [-r <rule_id>]
             [-d] [-sid SID] [-H <host>] [-P <port>]
@cyw3
Copy link
Collaborator

cyw3 commented Jan 12, 2022

麻烦更新下CodeAnalysis代码库，执行./compose_init.sh，然后重试下

@wushifeng
Copy link
Author

更新后的确可以运行的，不再提示Cobra的问题
不过运行一直不结束，无法看结果呢？

2022-01-13 08:36:48,356-INFO-util.logutil: Task_36 is done.
2022-01-13 08:36:48,718-INFO-util.logutil: Task_37 (PMD) starts ...
2022-01-13 08:36:58,830-INFO-util.logutil: Task_37 is done.
2022-01-13 08:36:59,199-INFO-util.logutil: Task_38 (Cpd) starts ...
2022-01-13 08:37:09,312-INFO-util.logutil: Task_38 is done.
2022-01-13 08:37:09,682-INFO-util.logutil: Task_39 (Lizard) starts ...
2022-01-13 08:37:19,795-INFO-util.logutil: Task_39 is done.
2022-01-13 08:37:20,161-INFO-util.logutil: Task_40 (CodeCount) starts ...
2022-01-13 08:37:30,267-INFO-util.logutil: Task_40 is done.
2022-01-13 08:37:31,130-INFO-util.logutil: Get code line data...
2022-01-13 08:37:31,131-INFO-util.logutil: ==================================================
2022-01-13 08:37:31,131-INFO-util.logutil: 本次扫描的代码量:
2022-01-13 08:37:31,131-INFO-util.logutil: code_line_num : 68570
2022-01-13 08:37:31,131-INFO-util.logutil: comment_line_num : 10108
2022-01-13 08:37:31,131-INFO-util.logutil: blank_line_num : 8472
2022-01-13 08:37:31,131-INFO-util.logutil: total_line_num : 87150
2022-01-13 08:37:31,131-INFO-util.logutil: filtered_code_line_num : 56048
2022-01-13 08:37:31,131-INFO-util.logutil: filtered_comment_line_num : 9458
2022-01-13 08:37:31,131-INFO-util.logutil: filtered_blank_line_num : 7525
2022-01-13 08:37:31,132-INFO-util.logutil: filtered_total_line_num : 73031
2022-01-13 08:37:31,132-INFO-util.logutil: filtered_lang_count : {'XML': {'file_num': 17, 'code_line_num': 1847, 'comment_line_num': 251, 'blank_line_num': 178, 'total_line_num': 2276}, 'TypeScript': {'file_num': 20, 'code_line_num': 2454, 'comment_line_num': 99, 'blank_line_num': 178, 'total_line_num': 2731}, 'Shell': {'file_num': 10, 'code_line_num': 112, 'comment_line_num': 30, 'blank_line_num': 39, 'total_line_num': 181}, 'SQL': {'file_num': 1, 'code_line_num': 14, 'comment_line_num': 13, 'blank_line_num': 6, 'total_line_num': 33}, 'gitignore': {'file_num': 5, 'code_line_num': 79, 'comment_line_num': 4, 'blank_line_num': 3, 'total_line_num': 86}, 'JavaServer Pages': {'file_num': 21, 'code_line_num': 942, 'comment_line_num': 4, 'blank_line_num': 110, 'total_line_num': 1056}, 'JavaScript': {'file_num': 54, 'code_line_num': 28019, 'comment_line_num': 3057, 'blank_line_num': 2099, 'total_line_num': 33175}, 'Plain Text': {'file_num': 2, 'code_line_num': 7, 'comment_line_num': 0, 'blank_line_num': 6, 'total_line_num': 13}, 'Java': {'file_num': 49, 'code_line_num': 2281, 'comment_line_num': 549, 'blank_line_num': 455, 'total_line_num': 3285}, 'TypeScript Typings': {'file_num': 19, 'code_line_num': 2689, 'comment_line_num': 4016, 'blank_line_num': 1454, 'total_line_num': 8159}, 'Markdown': {'file_num': 1, 'code_line_num': 27, 'comment_line_num': 0, 'blank_line_num': 4, 'total_line_num': 31}, 'JSON': {'file_num': 6, 'code_line_num': 1284, 'comment_line_num': 0, 'blank_line_num': 1, 'total_line_num': 1285}, 'CSS': {'file_num': 34, 'code_line_num': 16218, 'comment_line_num': 1418, 'blank_line_num': 2981, 'total_line_num': 20617}, 'HTML': {'file_num': 1, 'code_line_num': 59, 'comment_line_num': 0, 'blank_line_num': 11, 'total_line_num': 70}, 'Properties File': {'file_num': 5, 'code_line_num': 16, 'comment_line_num': 17, 'blank_line_num': 0, 'total_line_num': 33}}
2022-01-13 08:37:31,132-INFO-util.logutil: ==================================================
2022-01-13 08:37:32,679-INFO-util.logutil: query result from server ...

卡在最后一句，一直没有结果

config.ini配置

[COMMON]
TOOL_CONFIG_URL=https://hub.fastgit.org/TCATools/puppy-tools-config.git
PASSWORD_KEY=a6x4c7esudcv396w
DEBUG=
USE_LOCAL_TOOL=True

[SERVER_URL]
URL=http://localhost/server/main/

[FILE_SERVER]
URL=http://localhost/server/files/

@Lingghh
Copy link
Collaborator

Lingghh commented Jan 13, 2022

麻烦发一下analysis-worker的执行日志哈

docker-compose exec analysis-worker /bin/bash

tail -F log/codedog.log

@wushifeng
Copy link
Author

codedog.log中日志
-2022-01-12 18:26:22,925-DEBUG-asyncio: Using selector: EpollSelector
-2022-01-13 08:21:10,873-DEBUG-asyncio: Using selector: EpollSelector

root@591fe31dd2ae:/var/www/django/codedog/log# tail -5 celery.log 日志
-2022-01-13 10:52:53,895-WARNING-util.retrylib: method <download_file> exception: [116] download file exception: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cec04110>: Failed to establish a new connection: [Errno 111] Connection refused'))
-2022-01-13 10:52:53,895-INFO-util.retrylib: retrying method <download_file> on 3 time
-2022-01-13 10:52:58,901-INFO-util.fileserver: Downloading file from http://localhost/server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip to /var/www/django/codedog/tempdata/0a0f404ffcbd5c89ce0ae35d11b6983a27f391da/tmp_dir.zip
-2022-01-13 10:52:58,901-WARNING-urllib3.connectionpool: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72ceb60390>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip
-2022-01-13 10:53:02,904-WARNING-urllib3.connectionpool: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cec04d90>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip

  • docker-compose up的server日志截取
analysis-server_1  | [13/Jan/2022:10:50:17 +0800] <47> 172.18.0.11 'GET /api/projects/1/scans/12/ HTTP/1.1' 200 0.016400 603 -' 'python-urllib3/1.26.7'
main-server_1      | -2022-01-13 10:50:17,746-DEBUG-urllib3.connectionpool: http://analysis-server:8002 "GET /api/projects/1/scans/12/ HTTP/1.1" 200 603
main-server_1      | [13/Jan/2022:10:50:17 +0800] <53> 172.18.0.12 'GET /api/orgs/4dXNupGrLPa/teams/demo/repos/1/projects/1/scans/12/ HTTP/1.0' 200 0.050256 603 -' 'Python-urllib/3.7'
analysis-worker_1  | [2022-01-13 10:50:20,425: WARNING/ForkPoolWorker-2] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cebb8b10>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip
analysis-worker_1  | -2022-01-13 10:50:20,425-WARNING-urllib3.connectionpool: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cebb8b10>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip
analysis-worker_1  | [2022-01-13 10:50:20,426: WARNING/ForkPoolWorker-2] method <download_file> exception: [116] download file exception: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72ceb4c290>: Failed to establish a new connection: [Errno 111] Connection refused'))
analysis-worker_1  | -2022-01-13 10:50:20,426-WARNING-util.retrylib: method <download_file> exception: [116] download file exception: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72ceb4c290>: Failed to establish a new connection: [Errno 111] Connection refused'))
analysis-worker_1  | [2022-01-13 10:50:20,426: INFO/ForkPoolWorker-2] retrying method <download_file> on 2 time
analysis-worker_1  | -2022-01-13 10:50:20,426-INFO-util.retrylib: retrying method <download_file> on 2 time
main-server_1      | [13/Jan/2022:10:50:22 +0800] <53> 172.18.0.12 'POST /api/jobs/12/tasksbeat/ HTTP/1.0' 200 0.023438 82 -' 'Python-urllib/3.7'
analysis-worker_1  | [2022-01-13 10:50:25,432: INFO/ForkPoolWorker-2] Downloading file from http://localhost/server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip to /var/www/django/codedog/tempdata/0a0f404ffcbd5c89ce0ae35d11b6983a27f391da/tmp_dir.zip
analysis-worker_1  | -2022-01-13 10:50:25,432-INFO-util.fileserver: Downloading file from http://localhost/server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip to /var/www/django/codedog/tempdata/0a0f404ffcbd5c89ce0ae35d11b6983a27f391da/tmp_dir.zip
analysis-worker_1  | [2022-01-13 10:50:25,434: WARNING/ForkPoolWorker-2] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cebb8410>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip
analysis-worker_1  | -2022-01-13 10:50:25,434-WARNING-urllib3.connectionpool: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72cebb8410>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip
analysis-worker_1  | [2022-01-13 10:50:29,442: WARNING/ForkPoolWorker-2] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72ceb4c510>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip
analysis-worker_1  | -2022-01-13 10:50:29,442-WARNING-urllib3.connectionpool: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f72ceb4c510>: Failed to establish a new connection: [Errno 111] Connection refused')': /server/files/taskdata/projects/1/task_56/e6cab0ee741a11ecb7286651e7c0f790/tmp_dir.zip

猜测分析客户端配置的是localhost，而docker-compose中analysis肯定是不能使用localhost的

@wushifeng
Copy link
Author

另外客户端分析工具从github下载，国内使用的体验太差了
我在gitee上建了部分java相关的分析的镜像，地址如下：

https://gitee.com/organizations/TCATools/projects

客户端的配置设置为：
[COMMON]
; [必填]工具配置库git地址
#TOOL_CONFIG_URL=https://hub.fastgit.org/TCATools/puppy-tools-config.git
TOOL_CONFIG_URL=https://gitee.com/TCATools/puppy-tools-config.git

puppy-tools-config.git这个库中修改工具地址到https://gitee.com/TCATools
至少下载工具这块好很多

@wushifeng
Copy link
Author

原因已查到
分析客户端写成如下地址时不行
URL=http://localhost/server/main/
URL=http://localhost/server/files/

写如下这种具体的地址后，结果就可以上报了，估计代码有bug
URL=http://172.18.0.1/server/main/
URL=http://172.18.0.1/server/files/

但检查结果，查看需要修改的源代码时，又出问题【待会单独提个问题吧】
{"status_code":400,"code":1000,"msg":"未知错误: <class 'Exception'>:{method: cat_file, error_message: error: could not lock config file /var/www/django/codedog/sourcedirs/a12a4dc1c38f18898e86c1db745fb33fdc457ce3/.git/config: No such file or directory\nfatal: could not set 'core.repositoryformatversion' to '0'}"}

@yql70 yql70 mentioned this issue Jan 13, 2022
Closed
@yipeikk
Copy link

yipeikk commented Jan 14, 2022

你好 这边也遇到了这个问题 但是config.ini配置是正确的server地址 还是卡在 query result这一步

image

@cyw3
Copy link
Collaborator

cyw3 commented Jan 14, 2022

原因已查到 分析客户端写成如下地址时不行 URL=http://localhost/server/main/ URL=http://localhost/server/files/

写如下这种具体的地址后，结果就可以上报了，估计代码有bug URL=http://172.18.0.1/server/main/ URL=http://172.18.0.1/server/files/

但检查结果，查看需要修改的源代码时，又出问题【待会单独提个问题吧】 {"status_code":400,"code":1000,"msg":"未知错误: <class 'Exception'>:{method: cat_file, error_message: error: could not lock config file /var/www/django/codedog/sourcedirs/a12a4dc1c38f18898e86c1db745fb33fdc457ce3/.git/config: No such file or directory\nfatal: could not set 'core.repositoryformatversion' to '0'}"}

好的，麻烦单独提个issue哈

@cyw3 cyw3 closed this as completed Jan 14, 2022
This was referenced Jan 17, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wushifeng @Lingghh @cyw3 @yipeikk