Skip to content

feat: skill level, ability/image scraping, skills.csv, urls.csv - #4

Merged
AofeiLi-code merged 2 commits into
AofeiLi-code:mainfrom
bupt-lxc:feat/skill-level-ability-images
May 1, 2026
Merged

feat: skill level, ability/image scraping, skills.csv, urls.csv#4
AofeiLi-code merged 2 commits into
AofeiLi-code:mainfrom
bupt-lxc:feat/skill-level-ability-images

Conversation

@bupt-lxc

@bupt-lxc bupt-lxc commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Bug Fixes (previously PR #3)

  • 列表页解析:修复解析器以匹配实际 HTML 结构
  • 技能威力解析:原本所有技能的威力解析都有误,均解析为0,修正威力字段解析逻辑,去重技能列表

New Features

  • 技能等级parse_skills() 新增 level 字段,从 rocom_sprite_skill_level 解析 LV\d+
  • 图片爬取:爬取精灵立绘、技能图标、属性图标、特性图标、克制表图标,下载到 data/images/ 各子目录
  • urls.csv:记录所有图片 URL 及本地路径,按类型分类,边爬边更新,去重
  • skills.csv:全技能去重列表,列名对齐 skills_all.csv
  • 断点续爬:默认跳过已爬取的精灵,只爬新增内容
  • --force 参数:强制重爬所有精灵数据(图片已存在则跳过下载)

Test Plan

  • python rocom_scraper.py --limit 5 — 验证跳过逻辑
  • 全量爬取 465 只精灵,成功 415,跳过 50,失败 0
  • sprites.json 技能含 level 字段
  • urls.csv 边爬边写,五种图片类型均有记录
  • skills.csv 去重,列名正确
  • data/images/ 各子目录图片正常下载

@bupt-lxc
bupt-lxc force-pushed the feat/skill-level-ability-images branch from be14a45 to 7d10e03 Compare April 23, 2026 06:26
Kizzen983 added 2 commits April 23, 2026 14:34
…rce flag

- parse_skills: add level field (LV\d+ from rocom_sprite_skill_level)
- parse_sprite_detail: scrape ability icon, sprite image, matchup icons
- _add_url/_flush_urls: incremental urls.csv with per-type image download
- _save_skills_csv: deduplicated skills.csv (no 所属精灵 column)
- main: skip already-scraped sprites by default; --force clears and re-scrapes all
@bupt-lxc
bupt-lxc force-pushed the feat/skill-level-ability-images branch from 7d10e03 to c9b5258 Compare April 23, 2026 06:41
@AofeiLi-code
AofeiLi-code merged commit da860b3 into AofeiLi-code:main May 1, 2026
@bupt-lxc
bupt-lxc deleted the feat/skill-level-ability-images branch May 6, 2026 04:47
AofeiLi-code pushed a commit that referenced this pull request May 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants