Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Add a back end maintenance task for the crawler #1057
This is a first draft and I really need some help from here on I guess :)
This is how it looks at the moment:
leofeyer left a comment
I have made the necessary adjustments in aef6723. Here are some screenshots:
There are still some things which need to be fixed though:
Another thing that I am not happy with is using
Can we create the files in
Needs to be fixed in Escargot: terminal42/escargot#9
I disagree. In my tests it actually misses quite a lot with a depth of only 3. But we might configure 8 or so. Would that be okay?
Fixed in 68e636a.
Imho that log should go to the top right or so because it's a general log file and right now it looks as if it would belong to a certain subscriber.
Removed in 68e636a.
Description ----------- This is a first draft and I really need some help from here on I guess :) This is how it looks at the moment: <img width="1198" alt="Bildschirmfoto 2019-12-04 um 18 03 03" src="https://user-images.githubusercontent.com/481937/70164037-c140c200-16c0-11ea-9939-3072724a515a.png"> ToDo's: - [x] Bring back the FE member authentication. We need to improve this in a future version to have stateless authentication but for now, it should at least continue to work as before. - [x] Styling. It's all just some inline styles and not really pretty. I'm just really lacking the skills here. - [x] What configuration options should we provide? All of them according to the crawl command? Does it even make sense to let the users decide on concurrency etc.? - [x] Provide subscriber-specific logs (for level info+) Commits ------- e287095 First draft for the back end implementation 2f0d8b9 Implemented subscriber specific log files 7a5685d CS 32208f6 CrawlCommand CS 87fa4cf Discard back end configuration 5ca44b7 Re-implemented FE member auth as previously implemented 54f7a08 CS 9088730 Fix issue with timeout because of session deadlock 3ccffd8 Make sure the back end is never followed ef8158c Merge branch 'master' into feature/crawl-backend 08be7b0 Fix the coding style 3bdd8dc Converted DCA to Doctrine entity and also properly named it 9a5a9e3 Fixed labels aef6723 Adjust the back end implementation 68e636a Translated summary and warnings of all subscribers for the back end 76d33c5 Removed robots meta tags 45ce63e CS c968781 Move the debug log link next to the progress bar b2cec2c Move the CRAWL. labels into the default.xlf file and fix some minor CS issues