A dead-simple service that screenshots webpages, and sends the links to your email.
Build using Express, Bull and NodeMailer.
- Security: All webpages are opened both in sandboxed and _icognito mode of chromium.
- Scalability: Handles ~2.3k reqs/s on a single node. For load performance, see load testing.
- Resilience: Chromium instance automatically restarts on sudden crash without interrupting the job queue. Jobs that get affected are auto-retried.
- Reduced memory footprints: Puppeteer is used under the hood to capture screeshots. To reduce memory footprint, only one Chromium instance exists throughout the service lifetime irrespective of the number of screenshot workers that are running.
- Portability: Docker build (with
docker-compose) supported and can easily be scaled in any containerized environment. - Monitoring: Screenshot performance metric is currently scrapped with Prometheus. See metrics.
Demo application is currently hosted at: https://quickshots.herokuapp.com
See API Section on how to get started.
NOTE: Don't pass in any sensitive info in demo app.
You'll need to ensure you have the following installed:
- Clone the repo.
- Create a
.envfrom.env.example. - Get your Cloudinary API keys, and update env variables with prefix
CLOUDINARY_respectively. - Get a sample SMTP credentials at Ethereal, and update env variables with prefix
MAIL_SMTP_respectively. - Set
MAIL_DEFAULTtosmtp. - Set
REDIS_URLtoredis(only if you'll be running the app viadocker-composefor simplicity sake).
NOTE: If you want to start chromium in
--no-sandboxmode, simply specify the envLAUNCH_CHROMIUM_IN_SANDBOX=false. By default, it is set totrue.
Ensure you have docker-compose installed on your machine.
For production, to specify env variables via a file, you'll need to name the file
.env.prod
To serve the application on the default port 3000, simply run:
docker-compose up --buildThis should serve application on the following URL: http://127.0.0.1:3000
To run a development server:
- Run
npm i(if you have chrome installed, runnpm i --no-optional). - Update your environment variables.
- Run
npm run start:concurrenct(watches and restarts the server with nodemon)
Don't forget to update your
REDIS_URLto your local instance.
Captures screenshot of a webpage and sends to your mail
POST /screenshot
Body
{
"url": "https://google.com",
"email": "betsy81@ethereal.email"
}Get scraped prometheus metrics. Currently supported metrics incudes:
url_screenshots: Captures status of URL screenshots (success,failure,invalid).
GET /metrics
This service heavily relies on dependency injection, and all IOC declarations can be found here. To register:
- Services: see
services.ioc.ts - Controllers: see
controller.ioc.ts
How fast screenshots will be attended to, is heavily dependent on the no. of concurrent workers processing the screenshot jobs.
Currently, the screenshot queue runs by default with 5 workers. If you'll need to increase it, update the env flag QUEUE_WORKER_SCREENSHOT_AND_MAIL to your desired no. of workers.
NOTE: No. of workers equals the maximum no. of chromium tabs that can be opened at any point in time, t.
All tests are ran using Jest.
Load tests are currently carried out with k6. To run a basic HTTP load test:
- Ensure you have k6 installed.
- On terminal 1: Run the application (see starting up).
- On terminal 2: Run
npm run load-test.
The default load test uses the folllowing parameters:
- Virtual users: 1000
- Test iterations: 10000 (10000 request shared amongst 1000 users)
NOTE: In the future, there'll also exist plan to load test the overall queue performance.
To configure load testing paramaters, see supported CLI parameters for k6, and run as follows:
k6 run --vus 1000 <add more parameters here> ./test/load-testing/index.tsHere are the machine specs for the sample load testing shown below:
- MacBook Pro (13-inch, 2019, Four Thunderbolt 3 ports)
- 2.8 GHz Intel Core i7
- 16 GB 2133 MHz LPDDR3
If you see any way to improve this service, please kindly open a PR.