Directory structure not being recognized from XML keys #31
Comments
|
That's strange indeed. Could you provide me your index.html? |
|
Here is my index.html - note that I only changed the bucketUrl. In addition, I've found a separate way to get the listing I want, so this isn't very high-priority, but I figured it would still be nice to have answered in case someone else runs into a similar situation. Thank you!
<!-- S3 Bucket Explorer Version: 1.8.1 -->
<!DOCTYPE html>
<html lang="en" style="overflow-y: auto;">
<head>
<script>
const config = {
title: 'ZAP Results',
subtitle: 'results',
logo: 'https://avatars.githubusercontent.com/u/6716868?s=200&v=4',
favicon: 'https://avatars.githubusercontent.com/u/6716868?s=200&v=4',
primaryColor: '#236bb5',
bucketUrl: 'https://zap-results.com',
// If bucketUrl is undefined, this script tries to determine bucket Rest API URL from this file location itself.
// This will only work for locations like these
// * https://s3.BUCKET-REGION.amazonaws.com/BUCKET-NAME/index.html
// * http://BUCKET-NAME.s3-website-BUCKET-REGION.amazonaws.com/index.html
// * https://storage.googleapis.com/BUCKET-NAME/index.html
// * https://BUCKET-NAME.s3-web.BUCKET-REGION.cloud-object-storage.appdomain.cloud/
// If bucketUrl is set manually, ensure this is the bucket Rest API URL, e.g.
// * https://s3.BUCKET-REGION.amazonaws.com/BUCKET-NAME
// * https://storage.googleapis.com/BUCKET-NAME
// The URL should return an XML document with <ListBucketResult> as root element.
rootPrefix: undefined, // e.g. 'subfolder/'
keyExcludePatterns: [/^index\.html$/],
pageSize: 50,
bucketMaskUrl: undefined,
// If bucketMaskUrl is set file urls will be changed from ${bucketUrl}/${file} to ${bucketMaskUrl}/${file}
// bucketMaskUrl: 'https://example.org'
// => https://example.org/foo/bar.txt
// bucketMaskUrl: document.location.origin
// => ${document.location.origin}/foo/bar.txt
defaultOrder: 'name-asc' // (name|size|dateModified)-(asc|desc)
}
</script>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link id="favicon" rel="shortcut icon" />
<script src="https://unpkg.com/vue@2.6.14/dist/vue.min.js" integrity="sha384-ULpZhk1pvhc/UK5ktA9kwb2guy9ovNSTyxPNHANnA35YjBQgdwI+AhLkixDvdlw4" crossorigin="anonymous"></script>
<script>Vue.config.productionTip = false;</script>
<style>[v-cloak] {display: none}</style>
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.8.1/css/all.css" integrity="sha384-50oBUHEmvpQ+1lW4y57PTFmhCaXp0ML5d60M1M7uH2+nqUivzIebhndOJK28anvf" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdn.materialdesignicons.com/5.3.45/css/materialdesignicons.min.css" integrity="sha384-rJzKgf2UUnpzXvYOutlTiL8YYlEwrfhBmJI+ymqTQ67Dw3qJJeHA76pmK2IaTjuD" crossorigin="anonymous">
<script src="https://unpkg.com/buefy@0.9.10/dist/buefy.min.js" integrity="sha384-T58J1OGlL1z5XNzGTewXlzY/GUx9W3Nw7jUux1EJSA11VzG7YVKuzRTKo/lo/xRc" crossorigin="anonymous"></script>
<link rel="stylesheet" href="https://unpkg.com/buefy@0.9.10/dist/buefy.min.css" integrity="sha384-QtNuB2V0zXqYlM3LrzVmFc4U2pjfGzRfZbANB/BodC/oDS6aMw7CoDl4Me3X6UQS" crossorigin="anonymous">
<script src="https://unpkg.com/moment@2.29.1/min/moment.min.js" integrity="sha384-Uz1UHyakAAz121kPY0Nx6ZGzYeUTy9zAtcpdwVmFCEwiTGPA2K6zSGgkKJEQfMhK" crossorigin="anonymous"></script>
</head>
<body >
<div id="app" v-cloak :style="cssVars">
<!-- Header -->
<div class="level">
<div class="level-left" style="display: flex;">
<figure class="level-item image is-96x96" style="margin-right: 1.5rem;">
<img :src="config.logo" />
</figure>
<div>
<h1 class="title">{{config.title}}</h1>
<h2 class="subtitle">{{config.subtitle}}</h2>
</div>
</div>
</div>
<div class="container">
<!-- Navigation Bar -->
<div class="container is-clearfix">
<!-- Prefix Breadcrumps -->
<div class="buttons is-pulled-left">
<b-button v-for="(breadcrump, index) in pathBreadcrumps" v-bind:key="breadcrump.url"
type="is-primary" rounded
tag="a"
:href="breadcrump.url"
icon-pack="fas"
:icon-left="index == 0 ? 'folder' : ''"
target=""
:style="{ fontWeight: index == 0 ? 'bolder': ''}"
style="white-space: pre;"
>
<template>{{index > 0 ? breadcrump.name : '/'}}</template>
</b-button>
</div>
<!-- Paginating Buttons -->
<div v-show="nextContinuationToken || previousContinuationTokens.length > 0"
class="buttons is-pulled-right">
<b-button
type="is-primary" rounded
icon-pack="fas"
icon-left="angle-left"
@click="previousPage"
:disabled="previousContinuationTokens.length === 0"
>
</b-button>
<b-button
type="is-primary" rounded
icon-pack="fas"
icon-right="angle-right"
@click="nextPage"
:disabled="!nextContinuationToken"
>
</b-button>
</div>
</div>
<!-- Content Table -->
<b-table
:data="pathContentTableData"
:default-sort="config.defaultOrder.split('-')[0]"
:default-sort-direction="config.defaultOrder.split('-')[1]"
:mobile-cards="true"
>
<b-table-column
v-slot="props"
field="name"
label="Name"
sortable :custom-sort="sortTableData('name')"
cell-class="name-column"
>
<div style="display: flex; align-items: center;">
<b-icon
pack="far"
:icon="props.row.type === 'prefix' ? 'folder' : 'file-alt'"
class="name-column-icon"
>
</b-icon>
<div style="text-align: left;">
<b-button
type="is-text" rounded
tag="a"
:href="props.row.type === 'content' ? props.row.url : `#${props.row.prefix}`"
@click="document.activeElement.blur()"
style="text-align: left; white-space: pre-wrap;"
>
<template>{{ props.row.name }}</template>
</b-button>
<br>
<b-button
v-if="props.row.installUrl"
type="is-primary" rounded outlined
tag="a"
:href="props.row.installUrl"
class="name-column-install-button"
>
Install
</b-button>
</div>
</div>
<div
v-if="cardView && (props.row.size || props.row.dateModified)"
class="name-column-details"
>
<div>{{ formatBytes(props.row.size) }}</div>
<b-tooltip
:active="!!props.row.dateModified"
type="is-light"
size="is-small"
position="is-left"
animated
multilined
>
<div>{{ formatDateTime_relative(props.row.dateModified) }}</div>
<template v-slot:content>
<span>{{ formatDateTime_date(props.row.dateModified) }}</span>
<br>
<span>{{ formatDateTime_time(props.row.dateModified) }}</span>
</template>
</b-tooltip>
</div>
</b-table-column>
<b-table-column
v-slot="props"
field="size" numeric
label="Size"
sortable :custom-sort="sortTableData('size')"
centered width="128"
cell-class="size-column"
>
{{ formatBytes(props.row.size) }}
</b-table-column>
<b-table-column
v-slot="props"
field="dateModified"
label="Date Modified"
sortable :custom-sort="sortTableData('dateModified')"
centered width="256"
cell-class="modified-column"
>
<b-tooltip
:active="!!props.row.dateModified"
type="is-light"
size="is-small"
position="is-left"
animated
multilined
>
<div>{{ formatDateTime_relative(props.row.dateModified) }}</div>
<template v-slot:content>
<span>{{ formatDateTime_date(props.row.dateModified) }}</span>
<br>
<span>{{ formatDateTime_time(props.row.dateModified) }}</span>
</template>
</b-tooltip>
</b-table-column>
</b-table>
<!-- Paginating Buttons -->
<div class="container is-clearfix" style="margin-top: 1rem;">
<div v-show="nextContinuationToken || previousContinuationTokens.length > 0"
class="buttons is-pulled-right">
<b-button
type="is-primary" rounded
icon-pack="fas"
icon-left="angle-left"
@click="previousPage"
:disabled="previousContinuationTokens.length === 0"
>
</b-button>
<b-button
type="is-primary" rounded
icon-pack="fas"
icon-right="angle-right"
@click="nextPage"
:disabled="!nextContinuationToken"
>
</b-button>
</div>
</div>
</div>
<!-- Footer -->
<div class="footer-bucket-url">
<a :href="`${config.bucketUrl}?prefix=${config.rootPrefix}${pathPrefix}`">Bucket: {{ config.bucketUrl }}</a>
</div>
</div>
<script>
function escapeRegExp(text) {
// $& means the whole matched string
return text.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')
}
function escapeHTML(text) {
let div = document.createElement('div')
div.innerText = text
return div.innerHTML
}
function devicePlatform_iOS() {
return /iPad|iPhone|iPod/.test(navigator.platform) ||
(navigator.platform === 'MacIntel' && navigator.maxTouchPoints > 1)
}
</script>
<!-- Main Script -->
<script>
if (!config.bucketUrl) {
// try get bucket url by request parameter
config.bucketUrl = new URL(window.location).searchParams.get('bucket')
}
if (!config.bucketUrl) {
config.bucketUrl = window.location.href
}
// try adjusting bucket url to bucket rest api endpoint
let match
let type
if (!match) {
type = 'AWS'
// check for urls like https://s3.eu-central-1.amazonaws.com/example-bucket/index.html
match = config.bucketUrl.match(/(?<protocol>[^:]+):\/\/s3\.(?<region>[^.]+)\.amazonaws.com\/(?<name>[^/]+)/)
}
if (!match) {
type = 'AWS'
// check for urls like http://example-bucket.s3-website-eu-west-1.amazonaws.com/index.html
match = config.bucketUrl.match(/(?<protocol>[^:]+):\/\/(?<name>[^.]+)\.s3-website-(?<region>[^.]+)\.amazonaws\.com/)
}
if (!match) {
type = 'AWS'
// check for urls like http://example-bucket.s3-website.eu-central-1.amazonaws.com/index.html
match = config.bucketUrl.match(/(?<protocol>[^:]+):\/\/(?<name>[^.]+)\.s3-website\.(?<region>[^.]+)\.amazonaws\.com/)
}
if (!match) {
type = 'GCP'
// check for urls like https://storage.googleapis.com/example-bucket/index.html
match = config.bucketUrl.match(/(?<protocol>[^:]+):\/\/storage\.googleapis\.com\/(?<name>[^.]+)/)
}
if (!match) {
type = 'IBM'
// check for urls like http://example-bucket.s3-web.us-south.cloud-object-storage.appdomain.cloud/index.html
match = config.bucketUrl.match(/(?<protocol>[^:]+):\/\/(?<name>[^.]+)\.s3-web\.(?<region>[^.]+)\.cloud-object-storage\.appdomain\.cloud/)
}
if (match) {
switch (type) {
case 'AWS':
config.bucketUrl = `${match.groups.protocol}://s3.${match.groups.region}.amazonaws.com/${match.groups.name}`
break;
case 'GCP':
config.bucketUrl = `${match.groups.protocol}://storage.googleapis.com/${match.groups.name}`
break;
case 'IBM':
config.bucketUrl = `${match.groups.protocol}://${match.groups.name}.s3.${match.groups.region}.cloud-object-storage.appdomain.cloud/`
break;
}
}
console.log("Bucket REST API: " + config.bucketUrl)
config.rootPrefix = config.rootPrefix || ''
if (config.rootPrefix) {
if (!config.rootPrefix.endsWith('/')) {
config.rootPrefix += '/'
}
console.log("Bucket Root Prefix: " + config.rootPrefix)
}
document.title = config.title
document.getElementById('favicon').href = config.favicon
Vue.use(Buefy.default, {
defaultIconPack: 'fa'
})
new Vue({
el: '#app',
data: {
config, // defined in <head> section
pathPrefix: '',
pathContentTableData: [],
previousContinuationTokens: [],
continuationToken: undefined,
nextContinuationToken: undefined,
windowWidth: window.innerWidth
},
computed: {
cssVars() {
return {
'--primary-color': this.config.primaryColor
}
},
pathBreadcrumps() {
return `/${this.pathPrefix}`.match(/(?=[/])|[^/]+[/]?/g)
.map((pathPrefixPart, index, pathPrefixParts) => ({
name: decodeURI(pathPrefixPart),
url: '#' + pathPrefixParts.slice(0, index).join('') + pathPrefixPart
}))
},
cardView() {
return this.windowWidth <= 768
}
},
watch: {
pathPrefix() {
this.previousContinuationTokens = []
this.continuationToken = undefined
this.nextContinuationToken = undefined
this.refresh()
}
},
methods: {
moment,
previousPage() {
if (this.previousContinuationTokens.length > 0) {
this.continuationToken = this.previousContinuationTokens.pop()
this.refresh()
}
},
nextPage() {
if (this.nextContinuationToken) {
this.previousContinuationTokens.push(this.continuationToken)
this.continuationToken = this.nextContinuationToken
this.refresh()
}
},
async refresh() {
let listBucketResult
try {
if (!config.bucketUrl) {
throw Error("Bucket url is undefined!")
}
let bucketListApiUrl = `${config.bucketUrl}?list-type=2`
bucketListApiUrl += `&delimiter=/`
let bucketPrefix = `${config.rootPrefix}${this.pathPrefix}`
bucketListApiUrl += `&prefix=${bucketPrefix}`
if (config.pageSize) {
bucketListApiUrl += `&max-keys=${config.pageSize}`
}
if (this.continuationToken) {
bucketListApiUrl += `&continuation-token=${encodeURIComponent(this.continuationToken)}`
}
let listBucketResultResponse = await fetch(bucketListApiUrl)
let listBucketResultXml = await listBucketResultResponse.text()
listBucketResult = new DOMParser().parseFromString(listBucketResultXml, "text/xml")
if (!listBucketResult.querySelector('ListBucketResult')) {
throw Error("List bucket response does not contain <ListBucketResult> tag!")
}
} catch (error) {
this.$buefy.notification.open({
message: escapeHTML(error.message),
type: 'is-danger',
duration: 60000,
position: 'is-bottom'
})
throw error
}
let nextContinuationTokenTag = listBucketResult.querySelector("NextContinuationToken")
this.nextContinuationToken = nextContinuationTokenTag && nextContinuationTokenTag.textContent
let commonPrefixes = [...listBucketResult.querySelectorAll("ListBucketResult > CommonPrefixes")]
.map(tag => ({
prefix: tag.querySelector('Prefix').textContent
.replace(RegExp(`^${escapeRegExp(config.rootPrefix)}`), '')
}))
.filter(prefix => !config.keyExcludePatterns.find(pattern => pattern.test(prefix.prefix)))
.map(prefix => ({
type: 'prefix',
name: prefix.prefix.split('/').slice(-2)[0] + '/',
prefix: prefix.prefix
}))
let contents = [...listBucketResult.querySelectorAll("ListBucketResult > Contents")]
.map(tag => ({
key: tag.querySelector('Key').textContent,
size: parseInt(tag.querySelector('Size').textContent),
dateModified: new Date(tag.querySelector('LastModified').textContent)
}))
.filter(content => content.key !== decodeURI(this.pathPrefix))
.filter(content => !config.keyExcludePatterns.find(pattern => pattern.test(content.key)))
.map(content => {
if (content.key.endsWith('/') && !content.size) {
return {
type: 'prefix',
name: content.key.split('/')[0] + '/',
prefix: `${this.pathPrefix}${content.key}`
}
}
let url = `${(config.bucketMaskUrl || config.bucketUrl).replace(/\/*$/,'')}/${content.key}`
let installUrl = undefined
if (url.endsWith('/manifest.plist') && devicePlatform_iOS()) {
// generate manifest.plist install urls
installUrl = `itms-services://?action=download-manifest&url=${url.replace(/\/[^/]*$/,'')}/manifest.plist`
}
return {
type: 'content',
name: content.key.split('/').slice(-1)[0],
size: content.size,
dateModified: content.dateModified,
key: content.key,
url,
installUrl
}
})
this.pathContentTableData = [...commonPrefixes, ...contents]
},
sortTableData(columnName) {
return (rowA, rowB, isAsc) => {
// prefixes always first
if (rowA.type != rowB.type) {
return rowA.type === 'prefix' ? -1 : 1
}
const valueA = rowA[columnName]
const valueB = rowB[columnName]
if (valueA != valueB) {
if (valueA === undefined) {
return isAsc ? -1 : 1
}
if (valueB === undefined) {
return isAsc ? 1 : -1
}
return isAsc ?
(valueA < valueB ? -1 : 1) :
(valueA < valueB ? 1 : -1)
}
return 0
}
},
formatBytes(size) {
if (!size) {
return '-'
}
const KB = 1024
if (size < KB) {
return size + ' B'
}
const MB = 1000000
if (size < MB) {
return (size / KB).toFixed(0) + ' KB'
}
const GB = 1000000000
if (size < GB) {
return (size / MB).toFixed(2) + ' MB'
}
return (size / GB).toFixed(2) + ' GB'
},
formatDateTime_date(date) {
return date ? moment(date).format('ddd, DD. MMM YYYY') : '-'
},
formatDateTime_time(date) {
return date ? moment(date).format('hh:mm:ss') : '-'
},
formatDateTime_relative(date) {
return date ? moment(date).fromNow() : '-'
}
},
async mounted() {
window.onpopstate = () => this.pathPrefix = window.location.hash.replace(/^#\/?/, '')
window.onpopstate()
window.addEventListener('resize', () => {
this.windowWidth = window.innerWidth
})
this.refresh()
},
async beforeDestroy() {
window.removeEventListener('resize')
}
})
</script>
<style scoped>
body {
width: 100vw;
min-height: 100vh;
position: relative;
padding: 1.25rem 2.5rem 2.5rem 1.5rem;
background-color: #f5f5f5;
overflow-y: auto;
}
.button.is-primary {
background-color: var(--primary-color) !important;
}
.button.is-primary.is-outlined {
background-color: transparent !important;
border-color: var(--primary-color) !important;
color: var(--primary-color) !important;
}
.button.is-text {
padding: 0 !important;
height: auto !important;
text-decoration: none !important;
box-shadow: none !important;
background-color: unset !important;
user-select: text !important;
color: var(--primary-color) !important;
font-weight: 500;
}
.button.is-text:focus {
font-weight: bold;
}
.name-column-icon {
display: block;
text-align: left;
flex-basis: 1.5rem;
flex-grow: 0;
flex-shrink: 0;"
}
.name-column-install-button {
height: 0;
padding: 0.6rem;
margin-top: 0.4rem;
font-size: 0.8rem;
}
.name-column-details {
display: flex;
align-items: flex-end;
justify-content: center;
flex-shrink: 0;
min-width: 5rem;
font-size: 0.85rem;
line-height: 1.5rem;
flex-direction: column;
}
.footer-bucket-url {
position: absolute;
bottom: 0;
left: 0;
right: 0;
margin-bottom: 0.5rem;
font-size: small;
text-align: center;
color: darkgray;
}
.footer-bucket-url a {
color: inherit;
}
@media screen and (max-width: 768px) {
.name-column::before {
display: none !important;
}
.size-column,
.modified-column {
display: none !important;
}
}
</style>
</body>
</html> |
|
what the different way looks like? |
|
I just developed some HTML/JavaScript to send requests to pull the XML, then I did the parsing manually, no fancy libraries or anything like that. Didn't really have anything in common with the project here, sorry! |
|
I found the problem although I don't know why this is happening though. You should get a response like the following, with <ListBucketResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>data.openspending.org</Name>
<Prefix></Prefix>
<KeyCount>3</KeyCount>
<MaxKeys>50</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>index.html</Key>
<LastModified>2021-12-17T19:48:25.000Z</LastModified>
<ETag>"02c58fe9379dca33b143fc0f5af05ece"</ETag>
<Size>402</Size>
<StorageClass>DEEP_ARCHIVE</StorageClass>
</Contents>
<CommonPrefixes>
<Prefix>datasets/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>worldbank/</Prefix>
</CommonPrefixes>
</ListBucketResult> |
|
Makes sense, thank you! |
|
I think you don't use a S3 bucket api url. // If bucketUrl is set manually, ensure this is the bucket Rest API URL, e.g.
// * https://s3.BUCKET-REGION.amazonaws.com/BUCKET-NAME
// * https://storage.googleapis.com/BUCKET-NAME |
|
|
|
I'll add this api validation check to the bucket explorer script. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello, first of all thank you for creating this incredible tool! I'm running into a bit of an issue when my S3 bucket objects are being listed and hopefully you'll be able to help me.
I'm running a private S3 origin bucket for Cloudfront, and I can go to my designated URL and get the XML layout of all of the objects within, which should include directories such as 'HCU' and 'OWASPJuiceShop'. Note that some aspects were blurred for privacy reasons:
However, when I check the index.html page to see the reflected contents, this directory structure appears to be missing:
All I've modified in the example index.html you provide is my bucket URL (https://zap-results.com), and my search through the rest of the style changes doesn't seem to indicate that anything would remove these directory headings - any help would be greatly appreciated!
The text was updated successfully, but these errors were encountered: