Announcement: Crawly UI, first early prototype

[oltarasenko]

Hey people,

I have spent quite a bit of time prototyping a UI for Crawly. I think it's the next step we have to make in order to make Crawly visible in the Elixir and Web crawling space.

A good UI would help to convert web scraping into a process with clear create - test - use circles. Please have a glance on the early prototype here:
https://github.com/oltarasenko/crawly_ui

or test it here: http://18.216.221.122/jobs/1/items/

[Ziinc]

Really dig the new logo. The ui looks great, though i notice that its not responsive on mobile yet.

[oltarasenko]

@Ziinc Well, maybe walmart is a bad example. It has banned Crawly, without proxies it's a complex target. I am looking for a more simple but large site to showcase it

[Ziinc]

I think it is because there isn't rate limiting. It was scraping fine when I tried it.

Their anomaly detection probably kicks in due to the high hit rate.

[oltarasenko]

Yes it works atm. The crawl speed is quite low. Only 2 concurrent connections.

…

On Tue, Apr 28, 2020 at 6:46 PM Ziinc ***@***.***> wrote: I think it is because there isn't rate limiting. It was scraping fine when I tried it. Their anomaly detection probably kicks in due to the high hit rate. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAL3TP5FP2QS25J7O6IPKLRO4B5DANCNFSM4MRWKGPA> .

[oltarasenko]

Ok, now we have a PR. I will close this ticket now