Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use aws-sdk directly insted of through s3client (fixes #35) #39

Merged
merged 1 commit into from Apr 14, 2020
Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Use aws-sdk directly insted of through s3client (fixes #35)

The s3client module is unmaintained and indirectly depends on a
vulnerable version of the minimist module. Since it already
depends on the aws-sdk module, this change means we can remove a
number of dependencies without loss of functionality.
  • Loading branch information
fmarier committed Mar 28, 2020
commit 1ef5c5cad5f2cee8353476d02fcfa7724d4f7a4e

Some generated files are not rendered by default. Learn more.

@@ -3,8 +3,8 @@
"version": "6.0.0",
"description": "https everywhere ruleset builder for Brave",
"dependencies": {
"level": "^6.0.1",
"s3-client": "^4.4.2"
"aws-sdk": "^2.656.0",
"level": "^6.0.1"
},
"scripts": {
"build": "node scripts/preloadHTTPSE.js",
@@ -1,41 +1,34 @@
const fs = require('fs')
const s3 = require('s3-client')
const aws = require('aws-sdk')
const process = require('process')

const splitVersion = process.env.npm_package_version.split('.')
splitVersion.splice(2)
const dataFileVersion = splitVersion.join('.')

const client = s3.createClient({
maxAsyncS3: 20,
s3RetryCount: 3,
s3RetryDelay: 1000,
multipartUploadThreshold: 20971520,
multipartUploadSize: 15728640,
// See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property
s3Options: {}
const client = new aws.S3({
maxRetries: 3,

This comment has been minimized.

Copy link
@fmarier

fmarier Apr 13, 2020

Author Member

I wasn't able to find all of the old options since the API has changed significantly. It's not clear to me that they are needed though.

retryDelayOptions: { base: 1000 },
})

const uploadFile = (key, filePath, filename) => {
return new Promise((resolve, reject) => {
var params = {
localFile: filePath,
// See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property
s3Params: {
Bucket: 'https-everywhere-data',
Key: `${key}/${filename}`,
ACL: 'public-read'
}
Body: fs.createReadStream(filePath),
Bucket: 'https-everywhere-data',
Key: `${key}/${filename}`,
ACL: 'public-read'
}
var uploader = client.uploadFile(params)
process.stdout.write(`Started uploading to: ${params.s3Params.Key}... `)
uploader.on('error', function (err) {
console.error('Unable to upload:', err.stack, 'Do you have ~/.aws/credentials filled out?')
reject(new Error('Unable to upload'))
})
uploader.on('end', function (params) {
console.log('completed')
resolve()
console.log(`Started uploading to: ${params.Key}... `)
client.putObject(params, function(err, data) {
if (err) {
console.error('Unable to upload:', err.stack, 'Do you have ~/.aws/credentials filled out?')
reject(new Error('Unable to upload'))
} else {
console.log('completed')
resolve()
}
})
})
}
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.