Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use aws-sdk directly insted of through s3client (fixes #35) #39

Merged
merged 1 commit into from Apr 14, 2020
Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Some generated files are not rendered by default. Learn more.

@@ -3,8 +3,8 @@
"version": "6.0.0",
"description": "https everywhere ruleset builder for Brave",
"dependencies": {
"level": "^6.0.1",
"s3-client": "^4.4.2"
"aws-sdk": "^2.656.0",
"level": "^6.0.1"
},
"scripts": {
"build": "node scripts/preloadHTTPSE.js",
@@ -1,41 +1,34 @@
const fs = require('fs')
const s3 = require('s3-client')
const aws = require('aws-sdk')
const process = require('process')

const splitVersion = process.env.npm_package_version.split('.')
splitVersion.splice(2)
const dataFileVersion = splitVersion.join('.')

const client = s3.createClient({
maxAsyncS3: 20,
s3RetryCount: 3,
s3RetryDelay: 1000,
multipartUploadThreshold: 20971520,
multipartUploadSize: 15728640,
// See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#constructor-property
s3Options: {}
const client = new aws.S3({
maxRetries: 3,

This comment has been minimized.

Copy link
@fmarier

fmarier Apr 13, 2020

Author Member

I wasn't able to find all of the old options since the API has changed significantly. It's not clear to me that they are needed though.

retryDelayOptions: { base: 1000 },
})

const uploadFile = (key, filePath, filename) => {
return new Promise((resolve, reject) => {
var params = {
localFile: filePath,
// See: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property
s3Params: {
Bucket: 'https-everywhere-data',
Key: `${key}/${filename}`,
ACL: 'public-read'
}
Body: fs.createReadStream(filePath),
Bucket: 'https-everywhere-data',
Key: `${key}/${filename}`,
ACL: 'public-read'
}
var uploader = client.uploadFile(params)
process.stdout.write(`Started uploading to: ${params.s3Params.Key}... `)
uploader.on('error', function (err) {
console.error('Unable to upload:', err.stack, 'Do you have ~/.aws/credentials filled out?')
reject(new Error('Unable to upload'))
})
uploader.on('end', function (params) {
console.log('completed')
resolve()
console.log(`Started uploading to: ${params.Key}... `)
client.putObject(params, function(err, data) {
if (err) {
console.error('Unable to upload:', err.stack, 'Do you have ~/.aws/credentials filled out?')
reject(new Error('Unable to upload'))
} else {
console.log('completed')
resolve()
}
})
})
}
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.