Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
470241e
fix link filtering on ingest
matthewhanson Jan 30, 2020
4e68e32
add backup way to create_indices (if it cannot be done during deploym…
matthewhanson Jan 30, 2020
b1942d7
rename prepare payload to create_indices in ingest lambda
matthewhanson Jan 31, 2020
834fb08
fix index prep
matthewhanson Jan 31, 2020
f21c99f
await creation of indices
matthewhanson Jan 31, 2020
195ff0e
fix index creation
matthewhanson Jan 31, 2020
72e6e6b
bump version to 0.1.0-rc1
matthewhanson Jan 31, 2020
0e963d1
fix checking index exists
matthewhanson Jan 31, 2020
1a8fee1
properly set mappings
matthewhanson Jan 31, 2020
33ec9a4
add additional logger info
matthewhanson Jan 31, 2020
f02a36f
filter out hirarchical links only, not all others
matthewhanson Jan 31, 2020
db36e71
switch to npm from yarn
matthewhanson Feb 12, 2020
2392880
update sortby for STAC 0.9.0 final
matthewhanson Feb 27, 2020
fe7e86c
bump version to 0.1.0-rc2
matthewhanson Feb 27, 2020
d2a244b
fix sortby
matthewhanson Feb 27, 2020
34f9cbe
update backend to support changes to sortby extension
matthewhanson Feb 27, 2020
29d697e
add dead letter queue
matthewhanson Mar 9, 2020
c080b14
add subscription for SNS -> SQS
matthewhanson Mar 9, 2020
25e7e44
remove queue lambda (SNS goes straight to SQS)
matthewhanson Mar 9, 2020
3c8cab8
remove queue lambda
matthewhanson Mar 9, 2020
556e6d6
sns straight to queue
matthewhanson Mar 10, 2020
c7c99f3
add in lambda for subscribing to sns topics
matthewhanson Mar 10, 2020
d0e1a7e
change yarn to npm
matthewhanson Mar 10, 2020
4c375fd
add package-lock to repo
matthewhanson Mar 10, 2020
97a583e
update queue name for subscription lambda
matthewhanson Mar 10, 2020
0583cb0
update package scripts
matthewhanson Mar 10, 2020
f18a885
update lambda artifact
matthewhanson Mar 10, 2020
552fa92
update subscribe-to-sns lambda
matthewhanson Mar 10, 2020
ff09685
separate out collections and items index creation
matthewhanson Mar 20, 2020
720485b
fix redefined var
matthewhanson Mar 20, 2020
3ab7c4f
use aliases for names of collections and items indices
matthewhanson Mar 20, 2020
8e9ffbf
bump version to 0.1.0
matthewhanson Mar 20, 2020
52726a6
update CHANGELOG
matthewhanson Mar 20, 2020
b0595de
move environment variables up in config
matthewhanson Mar 20, 2020
18ec5cf
updated package-lock
matthewhanson Mar 20, 2020
6a903c0
updated README
matthewhanson Mar 20, 2020
6d95006
use npm for postinstall and build
matthewhanson Mar 20, 2020
e5a1770
add failed-ingest lambda to process dead letter queue (just logs for …
matthewhanson Mar 20, 2020
b64a801
remove unused package
matthewhanson Mar 20, 2020
ca6a330
update some dev dependenceies
matthewhanson Mar 20, 2020
5561bfc
bump node version
matthewhanson Mar 20, 2020
c667f90
update packages
matthewhanson Mar 20, 2020
faa4a4c
updated dependencies
matthewhanson Mar 20, 2020
9f3673c
revert es size
matthewhanson Mar 20, 2020
825b774
update github workflow
matthewhanson Mar 20, 2020
e9ae912
updated tests to work with new ava
matthewhanson Mar 20, 2020
d269a6b
remvoe serverless elasticsearch plugin
matthewhanson Mar 20, 2020
a521d5b
update README
matthewhanson Mar 20, 2020
9ceee04
rename var so no conflict
matthewhanson Mar 20, 2020
11344ff
add unofficial edit endpoint
matthewhanson Mar 20, 2020
2cc2e65
fix failing tests
matthewhanson Mar 21, 2020
6478201
add ingest error section to README
matthewhanson Mar 21, 2020
78ca1c0
log each failed record
matthewhanson Mar 21, 2020
ba77a17
update timeouts and sqs visibility timeout
matthewhanson Mar 21, 2020
fb3ceb3
updated package config
matthewhanson Mar 21, 2020
6d1ac5a
log failed SNS or plain messages
matthewhanson Mar 21, 2020
b42473e
add editItem capability
matthewhanson Mar 21, 2020
bb6b9f1
disable editing of items through API
matthewhanson Mar 21, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/push.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ jobs:
- name: install node
uses: actions/setup-node@master
with:
node-version: 10.15
node-version: 12.14

- name: yarn install
run: yarn install
- name: npm install
run: npm install

- name: yarn test
run: yarn test
- name: npm run test
run: npm run test

1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ build
dist
cloudformation.yml
ddd.js
package-lock.json
yarn.lock
lerna-debug.log
*.lerna_backup
Expand Down
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
12.13
12.14
6 changes: 4 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.

## [Unreleased]

## [0.0.1] - 2019-12-19
## [0.1.0] - 2020-03-20

Initial release, forked from [sat-api](https://github.com/sat-utils/sat-api/tree/develop)
Initial release, forked from [sat-api](https://github.com/sat-utils/sat-api/tree/develop).

Compliant with STAC 0.9.0

[Unreleased]: https://github.com/stac-utils/stac-api/compare/master...develop
[0.0.1]: https://github.com/stac-utils/stac-api/tree/0.0.1
119 changes: 65 additions & 54 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,40 @@

![](https://github.com/stac-utils/stac-api/workflows/Push%20Event/badge.svg)

Stac-api is a STAC compliant Rest API for searching and serving metadata for geospatial data (including but not limited to satellite imagery).
Stac-api is a STAC compliant Rest API for searching and serving metadata for geospatial data (including but not limited to satellite imagery). The STAC version supported by a given version of stac-api is shown in the table below. Additional information can be found in the [CHANGELOG](CHANGELOG.md)

The STAC version supported by a given version of stac-api is shown in the table below. Additional information can be found in the [CHANGELOG](CHANGELOG.md)

| sat-api | STAC |
| stac-api Version | STAC Version |
| -------- | ---- |
| 0.1.0 | 0.9.x |

## Deployed STAC APIs
The following APIs are deployed instances of stac-api:

| Name | Version | Description |
| -------- | ---- | ---- |
| [Earth Search](https://earth-search-v0.aws.element84.com/) | 0.9.0 | Catalog of some AWS Public Datasets |


## Usage

Stac-api is a RESTful API that returns JSON, see the [documentation](http://stac-utils.github.io/stac-api), or the /api endpoint which is a self-documenting OpenAPI document. Here are some additional tools that might prove useful:

- [JSONView Chrome Extension](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc?hl=en): Useful for exploring the API in the browser.
- [sat-search](https://github.com/sat-utils/sat-search): A Python client library and CLI for searching a STAC compliant API
- [sat-fetch](https://github.com/sat-utils/sat-fetch): A Python client library and CLI for fetching areas of interest and creating stacked clipped imagery from a STAC catalog that has Cloud-Optimized GeoTiff (COG) assets.

The following APIs are deployed APIs using stac-api.

## Deployment

This repository contains Node libraries for running the API, along with a [serverless](https://serverless.com/) configuration file for deployment to AWS.

To deploy:
To create your own deployment of stac-api, first clone the repository:

```
$ yarn
$ yarn build
$ yarn run deploy
$ git clone https://github.com/stac-utils/stac-api.git
$ cd stac-api
```

### Environment variables

There are some environment variables used in the code. Some do not have defaults and must be set.
There are some settings that should be reviewed and updated as needeed in the [serverless config file](serverless.yml), under provider->environment:

| Name | Description | Default Value |
| ---- | ----------- | ------------- |
Expand All @@ -37,62 +44,66 @@ There are some environment variables used in the code. Some do not have defaults
| STAC_TITLE | Title of this catalog | STAC API |
| STAC_DESCRIPTION | Description of this catalog | A STAC API |
| STAC_DOCS_URL | URL to documentation | [https://stac-utils.github.io/stac-api](https://stac-utils.github.io/stac-api) |
| STAC_API_URL | The root endpoint of this API to use for links | Inferred from request |
| ES_BATCH_SIZE | Number of records to ingest in single batch | 500 |
| LOG_LEVEL | Level for logging (CRITICAL, ERROR, WARNING, INFO, DEBUG) | INFO |
| STAC_API_URL | The root endpoint of this API | Inferred from request |

After reviewing the settings, build and deploy the project.

## Development
```
$ npm install
$ npm run build
$ npm run deploy
```

The latest released version is on the [master branch](https://github.com/sat-utils/sat-api/tree/master), and the latest development version is on the [develop](https://github.com/sat-utils/sat-api/tree/develop) branch.
This will create a CloudFormation stack in the `us-west-2` region called `stac-api-dev`. To change the region or the stage name (from `dev`) provide arguments to the deploy command (note the additional `--` in the command, required by `npm` to provide arguments):

### Building local version
```
$ npm run deploy -- --stage mystage --region eu-central-1
```

# Install dependencies in package.json
$ yarn
Once deployed there is one final step - creating the indices and mappings in Elasticsearch. Invoke the Lambda function (either through the AWS Console or the AWS CLI) with a payload of:

# Run the build command in each of the packages (runs webpack)
$ yarn build
```
{
"create_indices": true
}
```

# To continually watch and build source files
$ yarn watch
Stac-api is now ready to ingest data!

# To run tests for all packages
$ yarn test
## Ingesting Data

### Building API docs
STAC Collections and Items are ingested by the `ingest` Lambda function, however this Lambda is not invoked directly by a user, it consumes records from the `stac-api-<stage>-queue` SQS. To add STAC Items or Collections to the queue, publish them to the SNS Topic `stac-api-<stage>-ingest`.

# To build API docs from the api spec
$ yarn build-api-docs
STAC Collections should be ingested before Items that belong to that Collection. Items should have the `collection` field populated with the ID of an existing Collection.

## About
### Subscribing to SNS Topics

[stac-api](https://github.com/stac-utils/stac-api) was forked from [stac-api](https://github.com/sat-utils/sat-api). The stac-api is for STAC versions 0.9.0+, while sat-api exists for versions of STAC prior to 0.9.0.
Stac-api can also be subscribed to SNS Topics that publish complete STAC Items as their message. This provides a way to keep stac-api up to date with new data. Use the AWS Lambda console for the function `stac-api-<stage>-subscibe-to-sns` to subscribe to an SNS Topic for which you have the full ARN and permission to subscribe to. This could be an SNS Topic you created yourself to publish STAC records to, or a publicly available one, such as for [Sentinel](https://github.com/sat-utils/sat-stac-sentinel).

### Unit Tests
```
$ yarn
$ yarn test
```
### Ingest Errors

Errors that occur during ingest will end up in the dead letter processing queue, where they are processed by the `stac-api-<stage>-failed-ingest` Lambda function. Currently all the failed-ingest Lambda does is log the error, see the CloudWatch log '/aws/lambda/stac-api-<stage>-failed-ingest' for errors.

## Development

The latest released version is on the [master branch](https://github.com/sat-utils/sat-api/tree/master), and the latest development version is on the [develop](https://github.com/sat-utils/sat-api/tree/develop) branch.

### Integration Tests
Navigate to the integration directory
```
$ cd ./tests/integration
```
Use the environment variable `DOCKER_NAME` to set your Docker host name.
Normally `localhost`.
```
$ export DOCKER_NAME=localhost
```
The AWS-SDK library also requires fake key fields to create a connection so set.
```
$ export AWS_ACCESS_KEY_ID=none
```
```
$ export AWS_SECRET_ACCESS_KEY=none
```
To run the tests
# Install dependencies in package.json
$ npm install

# Run the build command in each of the packages (runs webpack)
$ npm run build

# To run tests for all packages
$ npm run test

# To build API docs from the api spec
$ npm run build-api-docs
```
$ ./runIntegration.sh
```

## About

[stac-api](https://github.com/stac-utils/stac-api) was forked from [stac-api](https://github.com/sat-utils/sat-api). The stac-api is for STAC versions 0.9.0+, while sat-api exists for versions of STAC prior to 0.9.0.
Loading