A free web crawler on top of the incredibile AdonisJS Framework
Things wanting to get done.
- Working draft
- Recursive search
- Pause and continue button
- Normalize links by using the server response
- Normalize links by removing trailing hashes
- Convert links inside
CrawlerControllerto models - Set own regex for extracting data
- Write tests
Michael Czechowski mail@dailysh.it
Dependencies and external libraries are listed in the package.json
Adonis Crawler - A free web crawler on top of AdonisJs Framework
Copyright (C) 2017 Michael Czechowski
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.