Turn any website to serverless API (support SPA!)
Branch: master
Clone or download
Latest commit 7df35b2 Jan 28, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
frontend add footer Jan 28, 2019
public add footer Jan 28, 2019
.editorconfig init Jan 23, 2019
.gitignore init Jan 23, 2019
.gitpod.yml add front-end dev setup to gitpod configuration Jan 28, 2019
.upignore init Jan 23, 2019
LICENSE Initial commit Jan 19, 2019
README.md automate dev setup using gitpod Jan 28, 2019
app.js release Jan 24, 2019
config.js release Jan 24, 2019
package-lock.json init Jan 23, 2019
package.json init Jan 23, 2019
up.json release Jan 24, 2019

README.md

CloudQuery

Turn any website to serverless API (with SPA support!)

Try online: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/

API

Example url: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/query?url=https://news.ycombinator.com/best&selectors=*:nth-child(1) > * > *:nth-child(2) > *:nth-child(2) > :nth-child(1),:nth-child(3) *:nth-child(1) > *:nth-child(3) > *:nth-child(1)

Query params:

  • url: url of the webpage you want to fetch
  • selectors: css selectors describing the elements interests you

Returns:

  • innerText: element innerText
  • href: element href

Run locally

npm i
npm start
# open http://localhost:3000

How to deploy your own CloudQuery to AWS lambda

1. Configuration

  • update profile(AWS Credential Profiles) in up.json to use your own aws account
  • update rateLimit in config.json to set your own rate limit (Default: 5 request/hour)

2. Deploy

  1. npm install
  2. Install up
  3. Input up in terminal and hit enter :)
  4. up url to see the API baseURL of your CloudQuery

How to Contribute

  1. Install nodemon and parcel locally
  2. Start backend by npm run local
  3. Start frontend by
        cd frontend
        npm install
        npm start
  4. The backend and frontend server will both auto restart when seeing code change
  5. After adding new feature or fixing bugs, execute npm run build in frontend folder, frontend code will be built and copied to public folder.

Develop online

You can run CloudQuery in a free online dev environment using Gitpod:

Open in Gitpod

Thanks

  • up for deploying serverless API to AWS with ease
  • serverless-chrome for running chrome on AWS lambda
  • finder for making it easy to select elements on webpage