Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(custom-file-in-bucket): Custom file names are also support for S3 output. #1129

Conversation

sergargar
Copy link
Member

Context

Custom file names were not applied to S3 bucket uploads. (flag -F was not applicable to -D/B)

Description

Include custom file names as well when exporting prowler reports to S3.

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@sergargar sergargar requested a review from a team as a code owner May 9, 2022 13:06
Copy link
Member

@jfagoagas jfagoagas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check comments and inspect the following:

copyToS3() {
  # Prowler will copy each format to its own folder in S3, that is for better handling 
  # and processing by Quicksight or others.

  # Also, check if -F was introduced
  if [ -n "${OUTPUT_FILE_NAME+x}" ]; then
    OUTPUT_PATH="${OUTPUT_FILE_NAME}"
  else
    OUTPUT_PATH="$OUTPUT_DIR/prowler-output-${ACCOUNT_NUM}-${OUTPUT_DATE}"
  fi

  for output_format in "${MODES[@]}";
  do
    case ${output_format} in
      csv)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_CSV}"
      ;;
      html)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_HTML}"
      ;;
      json)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_JSON}"
      ;;
      json-asff)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_ASFF}"
      ;;
      *)
        echo "$OPTRED ERROR!$OPTNORMAL - Invalid output format copying to S3. Use -h for help."
        exit 1
      ;;
    esac
  done
}

s3cp(){
  OUTPUT_PATH="${1}"
  EXTENSION="${2}"
  "${AWSCLI}" ${PROFILE_OPT} s3 cp "${OUTPUT_PATH}.${EXTENSION}" s3://"${OUTPUT_BUCKET}"/${EXTENSION}/ --acl bucket-owner-full-control
}

include/outputs_bucket Outdated Show resolved Hide resolved
include/outputs_bucket Outdated Show resolved Hide resolved
@sergargar
Copy link
Member Author

Check comments and inspect the following:

copyToS3() {
  # Prowler will copy each format to its own folder in S3, that is for better handling 
  # and processing by Quicksight or others.

  # Also, check if -F was introduced
  if [ -n "${OUTPUT_FILE_NAME+x}" ]; then
    OUTPUT_PATH="${OUTPUT_FILE_NAME}"
  else
    OUTPUT_PATH="$OUTPUT_DIR/prowler-output-${ACCOUNT_NUM}-${OUTPUT_DATE}"
  fi

  for output_format in "${MODES[@]}";
  do
    case ${output_format} in
      csv)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_CSV}"
      ;;
      html)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_HTML}"
      ;;
      json)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_JSON}"
      ;;
      json-asff)
        s3cp "${OUTPUT_PATH}" "${EXTENSION_ASFF}"
      ;;
      *)
        echo "$OPTRED ERROR!$OPTNORMAL - Invalid output format copying to S3. Use -h for help."
        exit 1
      ;;
    esac
  done
}

s3cp(){
  OUTPUT_PATH="${1}"
  EXTENSION="${2}"
  "${AWSCLI}" ${PROFILE_OPT} s3 cp "${OUTPUT_PATH}.${EXTENSION}" s3://"${OUTPUT_BUCKET}"/${EXTENSION}/ --acl bucket-owner-full-control
}

I love it! Thank you @jfagoagas

@sergargar sergargar requested a review from jfagoagas May 9, 2022 15:53
include/outputs_bucket Show resolved Hide resolved
Copy link
Member

@jfagoagas jfagoagas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!!!

A really good fix here.

@jfagoagas jfagoagas merged commit 210f44f into master May 11, 2022
@jfagoagas jfagoagas deleted the PRWLR-456-allow-prowler-to-use-f-custom-file-name-extension-and-d-custom-bucket branch May 11, 2022 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants