cdp-uploader

Core delivery platform Node.js Frontend Template.

Requesting access
Requirements
- Node.js
API
S3
- Object metadata on S3 Objects
Configuration
Local development
Docker
- Development image
- Production image
Licence
- About the licence

Requesting access

Note

An S3 bucket must have been created by the CDP Platform team prior to requesting cdp-uploader integration. Read the bucket documentation here on how to do this

To enable the cdp-uploader for your bucket, head to #cdp-support on Slack and copy, paste and update the following:

Hello CDP Support 👋 Please can I enable the cdp-uploader for my service:

bucket_name - The name of the bucket.

owning_service - The name of the service that owns this S3 bucket.

Thank you

Requirements

Node.js

Please install Node.js >= v18 and npm >= v9. You will find it easier to use the Node Version Manager nvm

To use the correct version of Node.js for this application, via nvm:

cd cdp-uploader
nvm use

API

POST /initiate

Example /initiate request:

{
  "redirect": "https://myservice.com/nextPage",
  "callback": "https://myservice.com/callback",
  "s3Bucket": "myservice",
  "s3Path": "scanned",
  "metadata": {
    "customerId": "1234",
    "accountId": "1234"
  }
}

Example /initiate download request:

{
  "downloadUrls": ["https://myservice.com/file"],
  "callback": "https://myservice.com/callback",
  "s3Bucket": "myservice",
  "s3Path": "scanned",
  "metadata": {
    "customerId": "1234",
    "accountId": "1234"
  }
}

HTTP Headers:

Header name	Description	Required
User-Agent	Identifier of the service that calls cdp-uploader	TBD

Body parameters:

Parameter name	Description	Required
redirect	URL to redirect to after file has been successfully uploaded. Cannot be used together with downloadUrls.	no
downloadUrls	List of URLs pointing to files that should be downloaded and scanned. Cannot be used together with redirect.	no
s3Bucket	S3 bucket that file will be moved to once the scanning is complete	yes
s3Path	'Folder' in bucket where scanned files will be placed	no
callback	Url that will be called once all files in upload have been scanned	no
metadata	Map of additional information related to upload	no
mimeTypes	List of accepted mimeTypes	no
maxFileSize	Maximum size in bytes that a file can be (10MB is 10 * 1000 * 1000)	no

Note

We will generate an uploadId for the upload and fileIds for files in the upload request. Files (objects) will be moved to destination bucket under the path uploadId/fileId which will be prefixed with the bucket path if it has been provided.

The /initiate endpoint now supports initiating uploads via download URLs. You can either specify a redirect or downloadUrls, but not both. When downloadUrls are provided, cdp-uploader downloads and scans the files asynchronously on behalf of the caller.

Example response

{
  "uploadId": "b18ceadb-afb1-4955-a70b-256bf94444d5",
  "uploadUrl": "/upload-and-scan/b18ceadb-afb1-4955-a70b-256bf94444d5",
  "statusUrl": "https://cdp-uploader/status/b18ceadb-afb1-4955-a70b-256bf94444d5"
}

Parameter name	Description	Required
uploadId	Identifier used for the upload	yes
uploadUrl	Url which must be used for the upload. If the initiate was a download request, this field will not be present	no
statusUrl	Endpoint that can be polled for status of upload	yes

POST /upload-and-scan/{uploadId}

Path Parameters

Parameter Name	Description
`uploadId`	Unique ID for the upload. The `uploadId` is provided via the `/initiate` call.

Description

The /upload-and-scan/{uploadId} endpoint supports two types of POST requests:

Browser uploads (multipart/form-data): For public-facing forms or user-facing applications where a file is submitted directly from the browser.
API uploads (binary or JSON body): For integrations from other government departments or data centres that need to programmatically upload files. These can use a direct POST request with an appropriate Content-Type header (e.g. application/octet-stream) and an optional x-filename header for metadata.

This flexibility allows simple integration with automated or API-based workflows as well as traditional web forms.

Example: Browser (multipart/form-data)

<form
  action="/upload-and-scan/b18ceadb-afb1-4955-a70b-256bf94444d5"
  method="post"
  enctype="multipart/form-data"
>
  <label for="file">File</label>
  <input id="file" name="file" type="file" />
  <button>Upload</button>
</form>

Example: API (binary upload)

curl -X POST "https://example.gov.uk/upload-and-scan/b18ceadb-afb1-4955-a70b-256bf94444d5"   -H "Content-Type: application/octet-stream"   -H "x-filename: report.pdf"   --data-binary "@report.pdf"

Response

Once the upload is accepted, the user (or API client) will be redirected to the redirect URL provided in the /initiate response. The CDP Uploader will then asynchronously perform a virus scan on the uploaded file.

For API integrations, this redirect can be ignored — frontend clients should prevent automatic redirect following (for example, by using fetch with redirect: 'manual' or an equivalent mechanism).

Instead, the integration can rely on one of the following approaches:

Callback (optional): If a callback URL is configured, the CDP Uploader will send a callback once the virus scan is complete, including the scan result and file metadata.
Polling: If no callback URL is configured, the tenant or API client can poll the status endpoint to check the progress and outcome of the upload.

GET /status/{uploadId}

The status API provides information about uploaded files, virus scan status and S3 location. The API is intended to be polled by the frontend services, it is not public and cannot be called direct from the browser.

Path Parameters

Parameter Name	Description
uploadId	Unique id for that upload. UploadId is provided via the `/initiate` call

Query Parameters

Parameter Name	Description
debug	set to 'true' to debug information. Currently contains initiate request payload

Note

Debug should not be used in production

Response Payload with Debug

{
  "debug": {
    "request": {
      "redirect": "http://localhost:3000/creatures/e90fee47-ead0-45c7-8319-4388cca9ebbc/upload-status-poller?uploadId=ba477d67-0005-4c26-a673-fa5139a2adf5",
      "s3Bucket": "cdp-example-bucket",
      "metadata": {
        "example-id": "id"
      }
    }
  },
  "uploadStatus": "ready",
  "metadata": {
    "example-id": "id"
  },
  "form": {
    "a-form-field": "some value",
    "a-file-upload-field": {
      "fileId": "9fcaabe5-77ec-44db-8356-3a6e8dc51b13",
      "filename": "dragon-b.jpeg",
      "contentType": "image/jpeg",
      "fileStatus": "complete",
      "contentLength": 11264,
      "checksumSha256": "bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=",
      "detectedContentType": "image/jpeg",
      "s3Key": "3b0b2a02-a669-44ba-9b78-bd5cb8460253/9fcaabe5-77ec-44db-8356-3a6e8dc51b13",
      "s3Bucket": "cdp-example-node-frontend"
    },
    "another-form-field": "foobazbar"
  },
  "numberOfRejectedFiles": 0
}

Response Payload without Debug

{
  "uploadStatus": "ready",
  "metadata": {
    "example-id": "id"
  },
  "form": {
    "a-form-field": "some value",
    "a-file-upload-field": {
      "fileId": "9fcaabe5-77ec-44db-8356-3a6e8dc51b13",
      "filename": "dragon-b.jpeg",
      "contentType": "image/jpeg",
      "fileStatus": "complete",
      "contentLength": 11264,
      "checksumSha256": "bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=",
      "detectedContentType": "image/jpeg",
      "s3Key": "3b0b2a02-a669-44ba-9b78-bd5cb8460253/9fcaabe5-77ec-44db-8356-3a6e8dc51b13",
      "s3Bucket": "cdp-example-node-frontend"
    },
    "another-form-field": "foobazbar"
  },
  "numberOfRejectedFiles": 0
}

Example response with downloadUrls

If a single download URL was provided:

{
  "uploadStatus": "ready",
  "metadata": {
    "customerId": "1234",
    "accountId": "1234"
  },
  "form": {
    "file": {
      "fileId": "16463a29-a040-4921-8c98-b1adf3ff09ec",
      "filename": "example-document.pdf",
      "contentType": "application/pdf",
      "downloadUrl": "https://my-bucket.s3.eu-west-2.amazonaws.com/scanned/example-document.pdf",
      "fileStatus": "complete",
      "contentLength": 204800,
      "checksumSha256": "czZ6B/jFbdYI+uhuGpnQ097MjOwdsW4s0kEYL6vwMMo=",
      "detectedContentType": "application/pdf",
      "s3Key": "2Fcb87c080-f5d0-49e5-9afe-712bab4f4ab4/16463a29-a040-4921-8c98-b1adf3ff09ec",
      "s3Bucket": "my-bucket"
    }
  },
  "numberOfRejectedFiles": 0
}

If multiple download URLs were provided:

{
  "uploadStatus": "ready",
  "metadata": {
    "customerId": "1234",
    "accountId": "1234",
    "some-metadata": "example"
  },
  "form": {
    "files": [
      {
        "fileId": "16463a29-a040-4921-8c98-b1adf3ff09ec",
        "filename": "document-1.pdf",
        "contentType": "application/pdf",
        "downloadUrl": "https://my-bucket.s3.eu-west-2.amazonaws.com/scanned/document-1.pdf",
        "fileStatus": "complete",
        "contentLength": 204800,
        "checksumSha256": "czZ6B/jFbdYI+uhuGpnQ097MjOwdsW4s0kEYL6vwMMo=",
        "detectedContentType": "application/pdf",
        "s3Key": "2Fcb87c080-f5d0-49e5-9afe-712bab4f4ab4/16463a29-a040-4921-8c98-b1adf3ff09ec",
        "s3Bucket": "my-bucket"
      },
      {
        "fileId": "4549f0dc-48d2-4bf3-955e-ae0fcef0ea41",
        "filename": "document-2.pdf",
        "contentType": "application/pdf",
        "downloadUrl": "https://my-bucket.s3.eu-west-2.amazonaws.com/scanned/document-2.pdf",
        "fileStatus": "complete",
        "contentLength": 104800,
        "checksumSha256": "bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=",
        "detectedContentType": "application/pdf",
        "s3Key": "2Fcb87c080-f5d0-49e5-9afe-712bab4f4ab4/4549f0dc-48d2-4bf3-955e-ae0fcef0ea41",
        "s3Bucket": "my-bucket"
      }
    ]
  },
  "numberOfRejectedFiles": 0
}

Scenario	Behaviour
`redirect` provided	Uploader waits for browser uploads - scanned asynchronously
`downloadUrls` provided	Files are downloaded and scanned automatically and asynchronously by cdp-uploader.
Both `redirect` and `downloadUrls` provided	Request is rejected with an error message (mutually exclusive).
Multiple `downloadUrls` provided	Response includes an array under `form.files`.
Single `downloadUrl` provided	Response includes a single `form.file` object.

Parameter Name	Description
uploaderStatus	Have all scans completed, can be `initiated`, `pending` or `ready`
metadata	Extra data and identified set by the requesting service in the /initialize call. Returned exactly as they were presented
form	An object representing each field in the multipart request. Text fields are preserved exactly as they were sent, file fields contain details about the file.
numberOfRejectedFiles	Total number of files that have been rejected by the uploader
debug.request	When set to true, the initiate request payload received by cdp-uploader

File field in form

Parameter Name	Description
fileId	uuid of the file.
filename	filename of file uploaded, if present
contentType	The mime type as declared in the multipart upload
fileStatus	`complete` or `rejected` if the virus scan has completed, `pending` if its still in progress
contentLength	Size of file in bytes
checksumSha256	SHA256 check sum of file recieved by cdp-uploader before uploading to S3 bucket
detectedContentType	The mime type as detected by the CDP-Uploader
s3Bucket	S3 bucket where scanned file is moved. Only set if file status is `complete`
s3Key	S3 Path where scanned file is moved. Includes path prefix if set. Only set when fileStatus is `complete`
hasError	`true/false` Only set to true if the file has been rejected or could not be delivered. Reason is supplied in `errorMessage` field.
errorMessage	Reason why file was rejected. Error message is based on GDS design guidelines and can be show directly to the end-user.

Error Handling

A file can be rejected by uploader for a number of reasons. When this happens no file will be delivered to the destination S3 bucket. A rejected file has the following data set:

fileStatus: rejected
hasError: true
errorMessage: string

The errorMessage field is a test description of why the file was rejected.

Cause	errorMessage
Virus detected	`The selected file contains a virus`
File is empty	`The selected file is empty`
File size exceeds max size (either set in the /init call or the uploaders max default 100M)	`The selected file must be smaller than $MAXSIZE`
File doesn't match the mime types set in the init call	`The selected file must be a $MIMETYPES`
Any server side error in CDP-Uploader	`The selected file could not be uploaded – try again`
Failed download	`The selected file could not be downloaded`

The messages are based on the GDS File Upload guidelines

The intention of the errorMessage field is that the content can be displayed directly to the end user.

Callback

If a callback URL was provided in the /initiate request, cdp-uploader will POST a JSON payload to that URL once all files in the upload have been scanned (and delivered or rejected). The payload is the same as the Status response (without debug).

The request is sent with Content-Type: application/json.

Callback payload

Field	Type	Description
`uploadStatus`	`string`	Always `"ready"` when the callback fires. All scans have completed.
`metadata`	`object`	The metadata object supplied in the `/initiate` request, returned exactly as provided. `undefined` if none was set.
`form`	`object`	An object representing each field in the multipart request (or download). See File fields in callback below.
`numberOfRejectedFiles`	`number`	Count of files that were rejected (virus, empty, too large, wrong mime type, download failure, or server error).

File fields in callback

Text form fields are preserved as-is. File fields are objects with the following properties:

Field	Type	Nullable	Description
`fileId`	`string`	no	UUID of the file, assigned by cdp-uploader.
`filename`	`string`	yes	Original filename from the multipart upload or derived from the download URL. Absent when no filename was provided.
`contentType`	`string`	yes	The MIME type as declared in the multipart upload (or from the `Content-Type` header for download URLs). Absent when not provided.
`detectedContentType`	`string`	yes	The MIME type detected by cdp-uploader from the file's magic bytes. `undefined` for file types that cannot be detected (e.g. plain text, CSV, JSON) or when the file was rejected before upload.
`contentLength`	`number`	yes	Size of the file in bytes as measured after upload to S3. `undefined` if the file was rejected before upload completed.
`checksumSha256`	`string`	yes	Base64-encoded SHA-256 checksum of the file. `undefined` if the file was rejected before upload completed.
`fileStatus`	`string`	no	`"complete"` if the file passed the virus scan and was delivered, `"rejected"` if the file was rejected for any reason.
`s3Bucket`	`string`	yes	Destination S3 bucket. Only set when `fileStatus` is `"complete"`.
`s3Key`	`string`	yes	S3 object key (includes `s3Path` prefix if one was set in `/initiate`). Only set when `fileStatus` is `"complete"`.
`hasError`	`boolean`	yes	`true` when the file has been rejected. `undefined` otherwise.
`errorMessage`	`string`	yes	Human-readable reason for rejection, suitable for displaying to end users (follows GDS File Upload guidelines). Only present when `hasError` is `true`.
`downloadUrl`	`string`	yes	The original download URL. Only present for uploads initiated via `downloadUrls`.

Example: successful upload callback

{
  "uploadStatus": "ready",
  "metadata": {
    "customerId": "1234"
  },
  "form": {
    "button": "upload",
    "file": {
      "fileId": "9fcaabe5-77ec-44db-8356-3a6e8dc51b13",
      "filename": "photo.jpeg",
      "contentType": "image/jpeg",
      "detectedContentType": "image/jpeg",
      "contentLength": 11264,
      "checksumSha256": "bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=",
      "fileStatus": "complete",
      "s3Key": "scanned/3b0b2a02-a669-44ba-9b78-bd5cb8460253/9fcaabe5-77ec-44db-8356-3a6e8dc51b13",
      "s3Bucket": "cdp-example-node-frontend"
    }
  },
  "numberOfRejectedFiles": 0
}

Example: rejected file (virus) callback

{
  "uploadStatus": "ready",
  "metadata": {
    "customerId": "1234"
  },
  "form": {
    "button": "upload",
    "file": {
      "fileId": "f45d0dd4-dd3f-4235-9c45-da2edd5c89fd",
      "filename": "malicious.jpeg",
      "contentType": "image/jpeg",
      "detectedContentType": "image/jpeg",
      "contentLength": 10503,
      "checksumSha256": "bng5jOVC6TxEgwTUlX4DikFtDEYEc8vQTsOP0ZAv21c=",
      "fileStatus": "rejected",
      "hasError": true,
      "errorMessage": "The selected file contains a virus"
    }
  },
  "numberOfRejectedFiles": 1
}

Callback delivery and retries

cdp-uploader expects the callback endpoint to return an HTTP 2xx status to acknowledge receipt.
If the callback fails (non-2xx response or network error), the message is not removed from the internal queue and will be retried automatically after the SQS visibility timeout.
Once acknowledged, duplicate deliveries are suppressed — the same callback will not be sent again.

Intended use

Frontend services will poll the /status endpoint after a user has uploaded a file.

The payload contains all of the fields submitted in the HTML form with the values preserved with the exception of any fields that contained files. These fields are replaced with an object showing if they passed the scan and where in S3 they can be accessed.

Frontend services are expected to validate the content of the payload and present any errors back to the user. Any files uploaded by a user will never be sent directly to the frontend service.

S3

Object metadata on S3 Objects

The objects placed in your buckets contain the following metadata fields and will be returned by AWS as response headers

Parameter Name	Description
Content-Type	AWS system defined metadata of what AWS deem the content type to be.
x-amz-meta-contenttype	What cdp-uploader deemed the content-type to be from inspecting the file.
x-amz-meta-encodedfilename	Name of uploaded file when it contained non ascii characters. This is RFC-2047 encoded. Optional.
x-amz-meta-filename	Name of uploaded file when it does not contain non ascii characters. x-amz-meta-encodedfilename will be returned instead when it does. Optional.
x-amz-meta-fileid	uuid of the file.
x-amz-meta-uploadid	Unique id for that upload. UploadId is provided via the `/initiate` call.

Configuration

Runtime environment variables required or optional.

Add these where relevant to:

App config

For configurations that are not sensitive and per staging and prod environments. https://github.com/DEFRA/cdp-app-config

Secrets

For sensitive config for staging and prod environments this can be self-serviced in your portal service page's secrets tab.

https://portal.cdp-int.defra.cloud/services/YOURSERVICE/secrets

Docker compose

For local docker compose setups add these to your setup, e.g. modify locally your compose/aws.env or similar.

Local dev env var

Set as environment variables locally. E.g. as export commands if using bash, or .envrc if using direnv

Configuration names

Config name	ENV_VAR	Default	Required	Purpose
`bucketsAllowlist`	CONSUMER_BUCKETS	[]	[x]	A comma separated list of buckets uploader can write to. Can not be empty if not in development mode.
`mockVirusScanEnabled`	MOCK_VIRUS_SCAN_ENABLED	false		Boolean. Useful in local development

There are several other configs, such as AWS details, SQS queue names, polling times etc. For more details and other service configuration look in src/config/index.js

Local development

Developing services that use the CDP-Uploader

If your service is going to use the CDP-Uploader to receive files you may want to start by running the uploader locally. The easiest way to do this is using docker compose.

Docker Compose

The CDP-Uploader project provide as base compose.yml file to get you started.

Copy compose.yml as well as the ./compose folder into your own project and running docker compose pull and then docker compose up.

This will start:

redis
localstack (a local AWS emulator)
cdp-uploader

It will also inject a start-up script into the localstack container that automatically creates the queues and buckets needed by the uploader as well as a test bucket named 'my-bucket'. If your service requires its own bucket you can add a line to start-localstack.sh script to create one.

aws --endpoint-url=http://localhost:4566 s3 mb s3://cdp-uploader-quarantine
aws --endpoint-url=http://localhost:4566 s3 mb s3://my-bucket

  ## Insert your bucket here, e.g.
aws --endpoint-url=http://localhost:4566 s3 mb s3://your-service-bucket

Note

The script sets the mock AWS region to be eu-west-2. See aws.env for the other localstack environment variables. If your service is also talking to S3 then it will need to use the same region and credentials when talking to localstack. The can be done by simply setting environment variables before running your sevices locally:

export AWS_REGION=eu-west-2
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test

If everything has worked as expected the CDP-Uploader will be available on localhost port 7337.

Any other supporting services can be added to the compose file as required.

Relative vs Absolute URLs

In a real environment relative urls work fine since all the services are behind the same host, however locally they're going to be running on different ports so relative redirect URLs won't work!

When run in development mode the cdp-uploader will convert relative redirect urls into absolute urls using the ' referer' header. This should make uploader behave more or less the same locally as it would in a real environment.

Test Harness (mock scanning)

When running locally the CDP-Uploader will be running with its mock virus scanner enabled.

This does not actually virus scan files, rather it simulates a response based on filename. If you submit a file with the word virus in the name it will be flagged as infected.

EICAR files and local development

The test harness does not support EICAR virus test files yet, so if you are submitting one and getting a CLEAN response back from the uploader's test harness, this is expected. In the real environments EICAR files will work.

Setup for developing CDP-Uploader

Install application dependencies:

npm install

Development

To run the application in development mode run:

npm run dev

Updating dependencies

To update dependencies, globally install https://www.npmjs.com/package/npm-check-updates. Then run the below script, run tests, test the application and commit the altered package.json and package-lock.json files. For more options around updates check the package docs.

ncu -i

Tests

Unit tests

To run unit tests:

npm test

End-to-end tests

To run end-to-end tests, you will need to have a few docker compose services running locally:

export AWS_REGION=eu-west-2
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test

docker compose pull
docker compose up -d localstack redis
npm run test:e2e

Smoke tests

To run smoke tests start up the cdp-uploader via docker compose:

cd cdp-uploader
docker compose pull
docker compose up

Then clone https://github.com/DEFRA/cdp-uploader-smoke-tests and run it:

cd cdp-uploader-smoke-tests
ENVIRONMENT=local npm run test

AWS CLI

Install the AWS CLI https://aws.amazon.com/cli/

AWS Local

awslocal is a wrapper around the AWS CLI that talks to your local localstack
You can install a PIP https://github.com/localstack/awscli-local But note it only works with the older v1 of AWS CLI

AWS local alias

Alternatively just alias it locally:
- E.g in fish

function awslocal
  aws --profile localstack $argv --endpoint-url http://localhost:4566
end

And add to your .aws/credentials

[localstack]
aws_access_key_id = test
aws_secret_access_key = test
region=eu-west-2

LocalStack

Either run localstack directly via docker and AWS CLI, or via the localstack CLI.

Docker

Run AWS LocalStack Docker container:

docker run --pull=always -d -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack:latest

Note the exposed endpoint is http://localhost:4566

Localstack CLI

Install LocalStack CLI
Run the CLI

localstack start

Note the exposed endpoint is https://localhost:4566

Setup local S3 buckets

You need local buckets setup in localstack

aws --endpoint-url=http://localhost:4566 s3 mb s3://my-bucket
aws --endpoint-url=http://localhost:4566 s3 mb s3://cdp-uploader-quarantine

The --endpoint-url http://localhost:4566 may not be needed depending on how your awslocal is set up. Also note depending on how your localstack is running the endpoint may be http or https.

List local buckets

aws --endpoint-url=http://localhost:4566 s3 list-buckets

View bucket contents

aws --endpoint-url=http://localhost:4566 s3 ls s3://cdp-uploader-quarantine
aws --endpoint-url=http://localhost:4566 s3 ls s3://my-bucket

Of view in your browser:

http://localhost:4566/cdp-uploader-quarantine/
http://localhost:4566/my-bucket/

Empty bucket contents

aws --endpoint-url=http://localhost:4566 s3 rm s3://cdp-uploader-quarantine --recursive
aws --endpoint-url=http://localhost:4566 s3 rm s3://my-bucket --recursive

Setup local queues

aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name cdp-clamav-results
aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name cdp-uploader-scan-results-callback.fifo --attributes "{\"FifoQueue\":\"true\",\"ContentBasedDeduplication\": \"true\"}"

Purge local queues

aws --endpoint-url=http://localhost:4566 sqs purge-queue --region eu-west-2 --queue-url http://localhost:4566/000000000000/cdp-clamav-results
aws --endpoint-url=http://localhost:4566 sqs purge-queue --region eu-west-2 --queue-url http://localhost:4566/000000000000/cdp-uploader-scan-results-callback.fifo

Setup local test harness

When running locally there is a built-in test harness to simulate scan results. This requires an extra setup step

aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name mock-clamav
aws --endpoint-url=http://localhost:4566 s3api put-bucket-notification-configuration\
    --bucket $BUCKET_NAME\
    --notification-configuration '{
                                      "QueueConfigurations": [
                                         {
                                           "QueueArn": "arn:aws:sqs:eu-west-2:000000000000:mock-clamav",
                                           "Events": ["s3:ObjectCreated:*"]
                                         }
                                       ]
	                                }'

When running from the IDE the test harness is enabled by default. It can be enabled/disabled via the MOCK_VIRUS_SCAN_ENABLED environment variable. The test harness will listen for files being uploaded to the quarantine bucket. When a file arrives, it checks if the original filename matches the regex set in MOCK_VIRUS_REGEX (defaults to checking if the file has 'virus' in the name). There is a short delay (set via MOCK_VIRUS_RESULT_DELAY) to simulate scan time before the response is sent.

Local JSON API

Whilst the APIs are being developed this app uses a local JSON mock API. To start this locally run:

npm run mockApi

Production

To mimic the application running in production mode locally run:

npm start

Npm scripts

All available Npm scripts can be seen in package.json To view them in your command line run:

npm run

Docker

Development image

Build:

docker build --target development --no-cache --tag cdp-uploader:development .

Run:

docker run -p 3000:3000 cdp-uploader:development

Production image

Build:

docker build --no-cache --tag cdp-uploader .

Run:

docker run -p 3000:3000 cdp-uploader

Licence

THIS INFORMATION IS LICENSED UNDER THE CONDITIONS OF THE OPEN GOVERNMENT LICENCE found at:

http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3

The following attribution statement MUST be cited in your products and applications when using this information.

Contains public sector information licensed under the Open Government license v3

About the licence

The Open Government Licence (OGL) was developed by the Controller of Her Majesty's Stationery Office (HMSO) to enable information providers in the public sector to license the use and re-use of their information under a common open licence.

It is designed to encourage use and re-use of information freely and flexibly, with only a few conditions.

Name		Name	Last commit message	Last commit date
Latest commit History 390 Commits
.github/workflows		.github/workflows
.husky		.husky
compose		compose
src		src
uploads		uploads
.browserslistrc		.browserslistrc
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.eslintrc.cjs		.eslintrc.cjs
.gitignore		.gitignore
.npmignore		.npmignore
.npmrc		.npmrc
.nvmrc		.nvmrc
.prettierignore		.prettierignore
.prettierrc.js		.prettierrc.js
Dockerfile		Dockerfile
LICENCE		LICENCE
README.md		README.md
babel.config.cjs		babel.config.cjs
compose.yml		compose.yml
jest.config.cjs		jest.config.cjs
nodemon.json		nodemon.json
package-lock.json		package-lock.json
package.json		package.json
postcss.config.js		postcss.config.js
stylelint.config.js		stylelint.config.js
tsconfig.json		tsconfig.json
webpack.config.js		webpack.config.js

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

cdp-uploader

Requesting access

Requirements

Node.js

API

POST /initiate

HTTP Headers:

Body parameters:

Example response

POST /upload-and-scan/{uploadId}

Path Parameters

Description

Example: Browser (multipart/form-data)

Example: API (binary upload)

Response

GET /status/{uploadId}

Path Parameters

Query Parameters

Response Payload with Debug

Response Payload without Debug

Example response with downloadUrls

File field in form

Error Handling

Callback

Callback payload

File fields in callback

Example: successful upload callback

Example: rejected file (virus) callback

Callback delivery and retries

Intended use

S3

Object metadata on S3 Objects

Configuration

App config

Secrets

Docker compose

Local dev env var

Configuration names

Local development

Developing services that use the CDP-Uploader

Docker Compose

Relative vs Absolute URLs

Test Harness (mock scanning)

EICAR files and local development

Setup for developing CDP-Uploader

Development

Updating dependencies

Tests

Unit tests

End-to-end tests

Smoke tests

AWS CLI

AWS Local

AWS local alias

LocalStack

Docker

Localstack CLI

Setup local S3 buckets

List local buckets

View bucket contents

Empty bucket contents

Setup local queues

Purge local queues

Setup local test harness

Local JSON API

Production

Npm scripts

Docker

Development image

Production image

Licence

About the licence

About

Topics

Resources

Packages