Repository navigation
v1.4.2
🚀 新增功能
- 企业全景测绘与自动建组下沉后端:天眼查与工信部查询合并重构为企业全景测绘(全景双轨 / 官方备案),支持企业工商全称自动检索匹配天眼查 GID 并升级双轨穿透;自动建组意图(
auto_group、auto_group_scan、探测策略与周期)由后端调度器消费,取代浏览器页面轮询,升级前在途任务经增量迁移补齐意图;在途与失联的updating_icp_task_id按任务终态自动晋升或清理。 - 普通扫描任务关联资产组:新建资产侦查任务支持可选「关联资产组」,扫描完成后自动同步至该组;目标派生数量如实预告(域名逐个建任务,IP / CIDR 合并为一个任务);任务列表新增资产组列与同步状态透出,同步入口统一经资产组选择器下发。
- 站点状态多选与 HTTP 状态码多选筛选:资产站点支持多 HTTP 状态码组合筛选(
status__in/status_code__in),非法输入直接报错以收敛查询范围;新旧单值条件并存时按交集收敛,结果不依赖参数顺序。 - 系统设置可搜索与凭据总览:系统设置顶部新增关键词搜索直达目标 Tab 并高亮,跨 Tab 凭据配置状态收敛为一行摘要(默认折叠),未保存改动常驻提示并支持跳转;系统更新页新增开源仓库与版本发布动态入口。
- 任务聚合态筛选与统计口径对齐:任务列表指标卡与筛选共用后端判定(
active为非终态、abnormal为error + stop),卡片数量与点击筛选结果数量保持一致;阶段状态统一标注为「进行中 · <阶段名>」。
🛠️ 性能与重构
- 系统设置信息架构重组与并发治理收口:设置页收敛为资产测绘与情报、字典与规则库、扫描与并发、通知与回调、系统版本与更新;GitHub 监控队列纳入并发治理并改用峰值模型推导推荐值,三项队列并发语义限定为 Worker 进程数,与单任务内发包并发度分开表述;保存时未传递字段保留库中现值,规避静默覆盖,热扩缩容按运行态进程数重试并回报实际收敛结果。
- GeoIP 库按需下载与命名卷剥离镜像:
ip2region/GeoLite2库文件不再随镜像分发,部署时由tools/fetch_geoip.py按需获取并校验后装入命名卷arl_geoip(无MAXMIND_LICENSE_KEY时 GeoLite2 跳过并降级运行);配套新增内存换出观察工具与队列 OOM 牺牲优先级区分。 - 配置模板与运行时配置分离:运行生效配置与入库模板分离,测试与验证脚本中的环境标识与凭据改由环境变量注入;构建上下文排除微服务源码副本与已入库构建产物,运行时截图目录停止跟踪。
- Nuclei 引擎能力缓存与启动探测合并:合并冗余启动探测并缓存引擎能力,降低重复探测开销。
🐛 问题修复与加固
- Nuclei 结果归属与失败可观测:Nuclei 结果保留目标端口,规避不同服务结果被合并;扫描进程失败时上报并清理异常后临时文件;测绘任务异常路径收敛终态判定与日志可观测性。
- 任务同步链路加固:任务同步改用统一范围校验并支持域名与 IP 任务,派发前原子认领规避重复下发,下发失败时置为异常并保留错误原因;任务详情数据集合修正并支持全局数据源回退;任务 ID 非法格式直接返回未找到,避免退化为系统异常。
- WIH 内置规则启用后生效:补齐启用后未生效的内置信息提取规则;外部资产文本按文字写入 XLSX,规避公式注入。
- OSINT 派发与取消语义收敛:按任务类型区分派发入参校验并补齐天眼查派发目标字段,取消不再误报为崩溃;删除任务前补齐派发确认与取消语义。
- MCP 自签部署可用性:自签部署下默认证书校验导致客户端不可用的问题收敛,客户端仅在同机回环时自动信任本地证书并保留链校验,远程部署经
ARL_CA_BUNDLE指定;双副本客户端保持一致。 - 前端交互收敛:企业全景测绘高级参数改为常驻平铺;危险操作默认值、统计口径与状态术语统一;资产删除目标集合修正。
🛡️ 安全与配置
- 推送通道 TLS 校验启用:Telegram、钉钉、飞书、企业微信推送启用 TLS 证书校验,收敛中间人风险。
- MongoDB 文件描述符上限提升:
docker-compose.prod.yml增设nofile上限至 65535,规避资产集合较多时并发查询触发官方警告;新增arl_geoip命名卷,存量持久化卷名保持不变。 - 数据库平滑演进:增量迁移补齐并发配置缺失字段(仅补缺失、不覆盖已保存值)、企业测绘自动建组意图(仅在途任务)与失联建组状态自愈;资产范围索引增补
updating_icp_task_id;全部步骤经单步容错隔离,单项异常不阻断启动。 - 文档口径收敛:API 调用约定下沉至
docs/API_GUIDE.md(中英双语),README 保留全局能力与快速入口;收敛与代码实现不符的能力宣称;资产组术语统一,社区协作模板补齐;证书与凭据忽略规则补齐并验证生效。
🚀 New Features
- Enterprise panorama recon with backend-driven auto-grouping: Tianyancha and MIIT ICP lookups merged into enterprise panorama recon (panorama dual-track / official filing). Registered company names auto-match Tianyancha GIDs and upgrade to dual-track pivoting; auto-grouping intents (
auto_group,auto_group_scan, scan policy and interval) are consumed by the backend scheduler instead of browser polling, with in-flight tasks backfilled by incremental migration; staleupdating_icp_task_idmarkers are promoted or cleaned up according to task terminal states. - Asset-group association for ordinary scan tasks: the new recon task form offers an optional related asset group, with results synced to it on completion; fan-out is previewed honestly (one task per domain, merged task for IP/CIDR); the task list shows the linked group and sync state, with sync dispatched through a unified group selector.
- Multi-select HTTP status filtering: site assets support combined multi-status filtering (
status__in/status_code__in); invalid input fails fast to narrow query scope; coexisting legacy single-value conditions intersect, independent of parameter order. - Searchable system settings with credential overview: keyword search jumps to target tabs with highlighting; cross-tab credential states collapse into one summary line (folded by default); unsaved changes stay visible with jump links; the update page adds open-source repo and release links.
- Aggregate status filters aligned with statistics: task-list metric cards and filters share backend semantics (
active= non-terminal,abnormal=error + stop), keeping card counts equal to filtered results; phase states render uniformly as "进行中 · <phase>".
🛠️ Performance & Refactoring
- Settings IA regroup with concurrency governance: settings converge into recon intel, dictionaries, scan & concurrency, notifications, and version & update; the GitHub monitor queue joins concurrency governance with peak-model recommendations; the three queue values are defined as Worker process counts, distinct from per-task blasting concurrency; omitted fields keep stored values on save to avoid silent overwrites; pool scaling retries against runtime process counts and reports observed convergence.
- On-demand GeoIP with image decoupling:
ip2region/GeoLite2databases no longer ship in images;tools/fetch_geoip.pyfetches and verifies them into thearl_geoipnamed volume at deploy time (GeoLite2 skipped with degraded service withoutMAXMIND_LICENSE_KEY); memory-swap observation tooling and per-queue OOM priorities added. - Config template / runtime config split: effective runtime config separated from committed templates; environment markers and credentials in test scripts injected via environment variables; microservice sources and committed build artifacts excluded from build context; runtime screenshot dir untracked.
- Nuclei engine capability caching: redundant startup probes merged with cached engine capabilities, reducing repeated probe overhead.
🐛 Fixes & Hardening
- Nuclei result attribution and failure visibility: results keep target ports so findings from different services are not merged; scan process failures are reported with temp files cleaned up; recon abnormal paths converge on terminal-state and log observability.
- Task sync chain hardening: sync uses unified scope validation for domain and IP tasks, atomic claim before dispatch to avoid duplicate delivery, error state with cause preserved on dispatch failure; task-detail data namespaces corrected with global source fallback; malformed task IDs return not-found instead of degrading to system errors.
- WIH builtin rules take effect: missing builtin extraction rules backfilled so enabling them works; external asset text written as literal XLSX cells to block formula injection.
- OSINT dispatch/cancel semantics: dispatch params validated per task type with Tianyancha target field backfilled; cancellation no longer misreported as crashes; dispatch confirmation and cancellation semantics added before task deletion.
- MCP self-signed deployment availability: default certificate verification breaking clients on self-signed deployments mitigated — local certificates auto-trusted on loopback only with chain verification kept, remote deployments via
ARL_CA_BUNDLE; both client copies kept in sync. - Frontend interaction convergence: panorama recon advanced params always visible; dangerous-operation defaults, statistics semantics, and status terms unified; asset deletion target namespaces corrected.
🛡️ Security & Configuration
- TLS verification for push channels: Telegram, DingTalk, Feishu, and WeCom push enable TLS certificate verification, narrowing MITM exposure.
- MongoDB fd limit raised:
docker-compose.prod.ymlsetsnofileto 65535 to avoid official warnings under concurrent queries over many collections; newarl_geoipnamed volume; existing persistent volume names unchanged. - Smooth DB evolution: incremental migrations backfill missing concurrency fields (fill-missing-only), in-flight recon auto-group intents, and stale grouping-state healing; scope index gains
updating_icp_task_id; every step isolated by single-step fault tolerance so one failure never blocks startup. - Doc wording convergence: API conventions moved to
docs/API_GUIDE.md(bilingual), README keeps global capabilities and quick entries; capability claims not matching implementation removed; asset-group terminology unified with community templates added; cert/credential ignore rules completed and verified.
What's Changed
- fix(security): 为 Telegram 推送启用 TLS 证书校验 by @jikdarren in #69
- perf(nuclei): 合并启动探测并缓存引擎能力 by @WeilianC in #63
- fix(task): 修正任务详情查询和删除的数据集合 by @WeilianC in #64
- fix(nuclei): 保留目标端口,避免不同服务的结果被合并 by @owl234 in #73
- fix(nuclei): 报告扫描进程失败并清理异常后的临时文件 by @owl234 in #74
- fix(wih): 补齐启用后未生效的内置信息提取规则 by @owl234 in #75
New Contributors
- @jikdarren made their first contribution in #69
Full Changelog: v1.4.1...v1.4.2