Heroku Link: https://go-webscraper.herokuapp.com/v1/amazon/product/asin/{ASIN}
Table View Sample Link: https://go-webscraper.herokuapp.com/v1/amazon/product/asin/B01644OCVS
Bullet View Sample Link: https://go-webscraper.herokuapp.com/v1/amazon/product/asin/B004QWYCVG
If live demo doesn't work, please try local with curl. This is an example with default config.
curl -H "Content-Type: application/json" -v http://localhost:4000/v1/amazon/product/asin/B00D89VK3Q
- Golang 1.11.x
- Redis 5.0 (stable)
- Docker (>= 17.12.0)
- Curl or Postman
GET /v1/amazon/product/asin/{asin}
-redispassord=""
-redishost=:6379
-grpcport=3000
-gatewayport=4000
Test with PostMan
GET https://go-webscraper.herokuapp.com/v1/amazon/product/asin/B004QWYCVG
Expected Product Info in JSON
{
"product": {
"asin": "B002QYW8LW",
"name": "Baby Banana Infant Training Toothbrush and Teether",
"categories": [
{
"name": "Baby Products",
"level": "1"
},
{
"name": "Baby Care",
"level": "2"
},
{
"name": "Pacifiers, Teethers & Teething Relief",
"level": "3"
},
{
"name": "Teethers",
"level": "4"
}
],
"ranks": [
{
"rank_info": "#24 in Baby ",
"level": "1"
},
{
"rank_info": "#1 in Baby Health Care Products",
"level": "2"
},
{
"rank_info": "#2 in Baby Teether Toys",
"level": "3"
}
],
"dimensions": [
"4.3 x 0.4 x 7.9 inches"
],
"created_at": "2019-04-23T20:18:01.155542Z"
}
}
Error message returned in JSON for the ASIN not found
{
"error": "Not Found",
"message": "Not Found",
"code": 2
}
Test REST API with curl
curl -H "Content-Type: application/json" -v https://go-webscraper.herokuapp.com/v1/amazon/product/asin/B004QWYCVG
Result
* Trying 54.208.229.218...
* TCP_NODELAY set
* Connected to go-webscraper.herokuapp.com (54.208.229.218) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Heroku, Inc.; CN=*.herokuapp.com
* start date: Apr 19 00:00:00 2017 GMT
* expire date: Jun 22 12:00:00 2020 GMT
* subjectAltName: host "go-webscraper.herokuapp.com" matched cert's "*.herokuapp.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
> GET /v1/amazon/product/asin/B004QWYCVG HTTP/1.1
> Host: go-webscraper.herokuapp.com
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: application/json
>
< HTTP/1.1 200 OK
< Server: Cowboy
< Connection: keep-alive
< Content-Type: application/json
< Grpc-Metadata-Content-Type: application/grpc
< Date: Tue, 23 Apr 2019 20:45:44 GMT
< Content-Length: 590
< Via: 1.1 vegur
<
* Connection #0 to host go-webscraper.herokuapp.com left intact
{"product":{"asin":"B004QWYCVG","name":"Native Unisex Kid's Jefferson Slip-On Sneaker","categories":[{"name":"Clothing, Shoes \u0026 Jewelry","level":"1"},{"name":"Girls","level":"2"},{"name":"Shoes","level":"3"},{"name":"Flats","level":"4"}],"ranks":[{"rank_info":"#5 in Clothing, Shoes \u0026 Jewelry","level":"1"},{"rank_info":"#1 in Baby Boys' Oxfords \u0026 Loafers","level":"2"},{"rank_info":"#1 in Men's Fashion Sneakers","level":"3"},{"rank_info":"#1 in Women's Fashion Sneakers","level":"4"}],"dimensions":["8.3 x 2.8 x 2 inches"],"created_at":"2019-04-23T20:44:20.953757697Z"}}%
go get github.com/rnidev/go-webscaper
Dockerhub link: https://hub.docker.com/r/devrni/go-webscraper
- Build from local
docker build -t go-webscraper .
- Pull from Dockerhub
docker pull devrni/go-webscraper
- Docker Compose
docker-compose -f ./deployments/docker-compose.yml up
- Build binary to ./build/
make build
- Build and Run binary locally
make run
- Run tests
make test
- Clean up tests and binary files
make clean
- Build For Linux
make build-linux
go-redis: Type-safe Redis client for Golang
Colly: Scraper and Crawler Framework for Golang http://go-colly.org/
grpc-go: The Go language implementation of gRPC. HTTP/2 based RPC
grpc-gateway: gRPC to JSON proxy generator following the gRPC HTTP spec
zap: Blazing fast, structured, leveled logging in Go. https://godoc.org/go.uber.org/zap
testify: A toolkit with common assertions and mocks for testing
1.12.3
This project is licensed under the terms of the MIT license.
Standard Project Layout for Go, by Kyle Quest
REST is not the Best for Micro-Services GRPC and Docker makes a compelling case, by Alex Punnen
REST vs. gRPC: Battle of the APIs, by Gigi Sayfan
Scraping the Web in Golang with Colly and Goquery, by Benjamin Congdon