Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update aws sdk and image uploader #7555

Merged
merged 22 commits into from
May 30, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b1f298f
add new s3 package
RileySeaburg May 10, 2024
88ef557
add new s3 package upload code
RileySeaburg May 10, 2024
123d3bb
add image to test upload
RileySeaburg May 10, 2024
01be580
fix build issue
RileySeaburg May 10, 2024
3d8f2bd
Images/Files processed [skip ci]
May 10, 2024
d70a80f
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
ToniBonittoGSA May 13, 2024
ea19f2b
Address Feedback
RileySeaburg May 15, 2024
83c9de3
## Update Gulp Task
RileySeaburg May 15, 2024
7edf200
Images/Files processed [skip ci]
May 15, 2024
56d7590
remove comment
RileySeaburg May 15, 2024
4dfbf4b
Merge branch 'main' of https://github.com/GSA/digitalgov.gov into rs-…
RileySeaburg May 15, 2024
c6988a7
addressed feedback
RileySeaburg May 17, 2024
86f369f
fix example env
RileySeaburg May 17, 2024
c768ab6
Fix security issues
RileySeaburg May 17, 2024
3f6a5d9
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
RileySeaburg May 23, 2024
41bccd0
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
ToniBonittoGSA May 23, 2024
2ec41a9
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
ToniBonittoGSA May 23, 2024
796f03c
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
ToniBonittoGSA May 24, 2024
d00754f
Refactored fileUpload to save to digitalgov/static directory
nick-mon1 May 29, 2024
4c51b0e
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
RileySeaburg May 29, 2024
86bed02
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
ToniBonittoGSA May 30, 2024
21b2b1e
Merge branch 'main' into rs-update-aws-sdk-and-image-uploader
ToniBonittoGSA May 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
78 changes: 30 additions & 48 deletions config/gulp/file-upload.js
RileySeaburg marked this conversation as resolved.
Show resolved Hide resolved
RileySeaburg marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,69 +1,51 @@
require("dotenv").config();
const { series, src } = require("gulp"),
vinylPaths = require("vinyl-paths"),
del = require("del"),
s3config = {
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
const gulp = require("gulp");
const awspublish = require('gulp-awspublish');
const del = require("del");
const vinylPaths = require("vinyl-paths");

// Create a new publisher using S3 options
const publisher = awspublish.create({
//region: 'your-region-id', // Change to your AWS region
params: {
Bucket: 'digitalgov', // Change to your bucket name
},
s3 = require("gulp-s3-upload")(s3config);
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
});

/**
* Uploads images in /processed folder to digitalgov s3 bucket
* TODO: Refactor both functions into one
* https://github.com/uswds/uswds-compile/blob/223fe67c335135c55cf1be9dead44e6d363e219d/gulpfile.js#L204-L232
*/
// Define custom headers
const headers = {
RileySeaburg marked this conversation as resolved.
Show resolved Hide resolved
'Cache-Control': 'max-age=315360000, no-transform, public',
};

function uploadImage() {
console.log("starting image upload");

return src("content/uploads/_working-images/processed/*")
.pipe(
s3(
{
Bucket: "digitalgov",
ACL: "public-read",
},
{
maxRetries: 5,
}
)
)
return gulp.src("content/uploads/_working-images/processed/*")
.pipe(awspublish.gzip({ ext: '.gz' }))
RileySeaburg marked this conversation as resolved.
Show resolved Hide resolved
.pipe(publisher.publish(headers))
.pipe(publisher.cache())
.pipe(awspublish.reporter())
.pipe(vinylPaths(del));
}

/**
* Uploads files in /to-process folder to s3 static file bucket
* TODO: Refactor both functions into one
*/

function uploadFile() {
console.log("starting file upload");

return src("content/uploads/_working-files/to-process/*")
.pipe(
s3(
{
Bucket: "digitalgov/static",
ACL: "public-read",
},
{
maxRetries: 5,
}
)
)
publisher.config.params.Bucket = "digitalgov/static"; // Change to your specific static files bucket
RileySeaburg marked this conversation as resolved.
Show resolved Hide resolved
return gulp.src("content/uploads/_working-files/to-process/*")
.pipe(awspublish.gzip({ ext: '.gz' }))
.pipe(publisher.publish(headers))
.pipe(publisher.cache())
.pipe(awspublish.reporter())
.pipe(vinylPaths(del));
}

/**
* Deletes working directories after uploading is complete
* @returns nothing
*/
function cleanup() {
console.log("cleanup");
return del([
"content/uploads/_working-images/**",
"content/uploads/_working-files/**",
]);
}
RileySeaburg marked this conversation as resolved.
Show resolved Hide resolved

exports.do = series(uploadImage, uploadFile, cleanup);
exports.do = gulp.series(uploadImage, uploadFile, cleanup);
12 changes: 12 additions & 0 deletions data/images/clock-iconcop.yml
RileySeaburg marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@

# https://s3.amazonaws.com/digitalgov/clock-iconcop.png
# Image shortcode: {{< img src=clock-iconcop >}}'
date : 2024-05-10 21:07:58 -0400
uid : clock-iconcop
width : 14
height : 15
format : png
credit :
caption :
alt :