Skip to content

imafish/crawler-go

Repository files navigation

crawler-go

TODO

  1. rule handling
  2. do not insert to queue if already exists
  3. page encoding handling
    1. handle unrecognized character
    2. encoding is considered the same accross a group, consider change this
  4. front-end rendering
  5. conditional rules
  6. logging
  7. refactoring
    1. manage exported/unexported symbols
    2. namings
    3. create necessary folders
  8. retry
  9. reports
  10. grab text from page
  11. filters
  12. post actions
    1. make Task.Execute() return a Result object, store the Result into group context
  13. Pause X seconds

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages