[FEATURE] Add a webhook/webservice for new url submissions

[ghost]

Hi guys,

Hope you are all well !

Is it possible to add a webservice or webhook to collect data from a website ?

Something like [POST] http://localhost:3000/collect/?url=example.com

Thanks for any insights or inputs on that question.

Cheers,
Luc Michalski

[popstas]

I think about it, it's time to implement :)

Spent one hour for dirty MVP, try it:

npm run server

Then open:

• http://localhost:3000?url=https://blog.popstas.ru&max-requests=10 - return json
• http://localhost:3000?url=https://blog.popstas.ru&max-requests=10&upload=1 - return link to uploaded results

I don't test it well, but assume that it can receive GET params with names as in command line.

[ghost]

Hi @popstas ,

Hope you are all well !

I made a quick PR #7 for dockerizing the server but it needs some work/discussions from our side.

Here are my remarks for the current state of the project:

• Use env variables for defining servers default options (eg. output dir)
• Update the frontend dockerfile to use the static generated files instead of running the dev version
• Create a queuing system for the server (redis or kafka) when receiving urls to collect
• Use Traefik ou Caddy for serving both frontend/backend in compose

And, here are my suggestions for more advanced features:

• Create a more convenient frontend/UI for browsing reports (by date)
• Create organization/groups/teams user management system for websites

Your project is great :-) and I ll be more than happy to help you to consolidate your hard work.

Thanks for any feedbacks on these points

Cheers,
Luc Michalski

[popstas]

@lucmichalski, you can try MVP with web interface:

1. site-audit-seo: npm run server
2. site-audit-seo-viewer: npm run dev
3. http://localhost:3000/scan

Demo:

[ghost]

that's incredibly awesome :-)

Maybe you should create a queuing system with redis for cron jobs.

[popstas]

Adapted for docker!

Back - e9917db
Front - viasite/site-audit-seo-viewer@87a1faa

Changes:

• Created images on Docker Hub, now you can just docker-compose up to startup back and front, without build (maybe you should comment build line in docker-compose.yml)

Backend:

• Backend port now 5301. 530 read as SEO :)
• Remove Redis (too early)
• Remove docker networks (simplify)
• Change base image to buildkite/puppeteer (easy to run puppeteer)
• Remove git clone and npm install -g site-audit-seo (less dependencies)
• Remove viewer Dockerfile to viewer repo, for docker build you should symlink front, see CONTRIBUTING.md
• Change reports directory to data/reports (data ignored by git, docker)
• Add route /reports, it serve static reports, when no --upload defined

Frontend:

• Frontend port now 5302
• Client generate link to viewer (previous server must know client url)
• Work scans without --upload (serve from local server) and with --upload (upload to our server site-audit.viasite.ru)

[ghost]

Hi @popstas ,

Hope you are all well !

That's really cool :-) Congrats for this great work !!!

I'll deploy it today on my server and will come back with some feedbacks if I had some.

Do you think that the InfluxDB/Grafana integration is easily feasible ?

Cheers,
Luc Michalski

[popstas]

InfluxDB integration not so easy, I am going to add memory queue, maybe Google auth in 2020.
Grafana will be in Jan-Feb 2021.

[ghost]

Tell me if you need some help :-)

[popstas]

Finished web service, launched at https://viasite.github.io/site-audit-seo-viewer/scan/

Reports:

• Filter panel open when query not empty
• Select buttons replaced for switch and select
• Welcome screen for zero scans users
• Grayscaled secondary controls
• Minify paddings

Scan:

• Server URL moved to "Settings" collapsed block
• Terminal window fits on first screen (any screen)
• Minify paddings
• Changed report filenames
• Add Google auth
• Add server state, scans queue

[ghost]

Hi @popstas ,

Sorry to re-open this issue but your demo is not working properly.

It cannot find the json data file:

Maybe you can spot something in your logs.

Cheers,
Luc Michalski

[popstas]

Server under development and eventually working wrong. Please try later.

I added main settings to scan form.