Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

文字化けの除去 #1

Closed
Sunwood-ai-labs opened this issue Jun 9, 2024 · 0 comments
Closed

文字化けの除去 #1

Sunwood-ai-labs opened this issue Jun 9, 2024 · 0 comments

Comments

@Sunwood-ai-labs
Copy link
Owner

サイトによっては文字化けするページがあるので、それの自動除去
Sunwood-ai-labs added a commit that referenced this issue Jun 10, 2024
@Sunwood-ai-labs
📝 (#1)[docs] スクレイピングオプションの説明を更新
233951b 
- メソッドの変更に伴い、READMEファイル内のコマンド例を更新し、正確な使用方法を反映しました。
Sunwood-ai-labs added a commit that referenced this issue Jun 10, 2024
@Sunwood-ai-labs
🐛 (#1)[fix] 文字化け検出機能を追加してスクレイピングの堅牢性を向上
cd2592c 
- 文字化けを自動検出し、不正なテキストが含まれるページを除外する新機能を実装。
- ASCII範囲外の文字が多い場合、内容が文字化けしているとみなし、処理から除外します。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Sunwood-ai-labs