Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] ass 0.15.0 #220

Draft
wants to merge 382 commits into
base: master
Choose a base branch
from
Draft

[WIP] ass 0.15.0 #220

wants to merge 382 commits into from

Conversation

tycrek
Copy link
Owner

@tycrek tycrek commented Jul 7, 2023

Checklist

  • I have read the Contributing Guidelines
  • I acknowledge that any submitted code will be licensed under the ISC License
  • I confirm that submitted code is my own work
  • I have tested the code, and confirm that it works

Environment

  • Operating System: Arch Linux x64 6.6.3
  • Node version: v20.9.0
  • npm version: 10.1.0

Description

🍑 Work-in-progress branch for ass 0.15.0 🍑

⚠️ This version will essentially be a full rewrite.

Want to test it out?

This version is currently usable! A lot of things are still missing (see the Roadmap below) but the core functions are working. For instructions, visit the new documentation!

At this time, there is no user system with 0.15.0. Once the setup is complete, you are ready to upload! If you encounter any issues, please join the Discord server for assistance.

Roadmap

  • New launch flow (host server without user config)
  • New web-based setup
    • Basic flow
    • User creation
    • Make it look prettier
  • API for users
    • admin: create
    • admin: modify
    • admin: delete
    • user: modify
    • user: self-delete
  • API for resources
  • User management
    • Admin dashboard
    • Token generation
  • Upload flow
    • Token auth
  • MySQL
  • PostgreSQL
  • MongoDB
  • Other SQL
  • S3
  • Migrate other features
    • Generators: Gfycat, zws, timestamp, random, nanoid, original
    • File size
    • File hashing
    • node-vibrant
    • Thumbnail files
    • Remove GPS data (https://github.com/tycrek/ass/blob/master/src/nightmare.ts)
    • Media strict mode (user-configurable)
    • Delete files
    • Webhooks
    • Direct Download (and Accept-Range/Content-Length headers)
  • Embeds
    • OpenGraph
    • oEmbed
  • User frontend
  • Embed data
    • File data
    • Headers
    • Server-side/global
  • Rate-limiting

@tycrek tycrek self-assigned this Jul 7, 2023
@tycrek tycrek added this to the 0.15.0 milestone Jul 7, 2023
@tycrek tycrek mentioned this pull request Jul 7, 2023
4 tasks
backend/routers/setup.ts Fixed Show fixed Hide fixed
backend/UserConfig.ts Fixed Show fixed Hide fixed
backend/routers/setup.ts Fixed Show fixed Hide fixed
backend/routers/setup.ts Fixed Show fixed Hide fixed
backend/routers/index.ts Fixed Show fixed Hide fixed
backend/routers/index.ts Fixed Show fixed Hide fixed
backend/routers/index.ts Fixed Show fixed Hide fixed
backend/routers/index.ts Fixed Show fixed Hide fixed
Comment on lines 26 to 70
router.post('/', async (req, res) => {

// Check user config
if (!UserConfig.ready) return res.status(500).type('text').send('Configuration missing!');

// Does the file actually exist
if (!req.files || !req.files['file']) return res.status(400).type('text').send('No file was provided!');

// Type-check the file data
const bbFile: BusBoyFile = req.files['file'];

// Prepare file move
const uploads = UserConfig.config.uploadsDir;
const timestamp = Date.now().toString();
const destination = `${uploads}${uploads.endsWith('/') ? '' : '/'}${timestamp}_${bbFile.filename}`;
// todo: S3

try {

// Move the file
await fs.move(bbFile.file, destination);

// Build ass metadata
const assFile: AssFile = {
fakeid: random({ length: UserConfig.config.idSize }), // todo: more generators
id: nanoid(32),
mimetype: bbFile.mimetype,
filename: bbFile.filename,
timestamp,
uploader: '0', // todo: users
save: { local: destination },
sha256: '0' // todo: hashing
};

log.debug('File saved to', assFile.save.local!);

// todo: save metadata

return res.type('json').send({ resource: `${req.ass.host}/${assFile.fakeid}` });
} catch (err) {
log.error('Failed to upload file', bbFile.filename);
console.error(err);
return res.status(500).send(err);
}
});

Check failure

Code scanning / CodeQL

Missing rate limiting High

This route handler performs
a file system access
, but is not rate-limited.
This route handler performs
a file system access
, but is not rate-limited.
This route handler performs
a file system access
, but is not rate-limited.
This route handler performs
a file system access
, but is not rate-limited.
This route handler performs
a file system access
, but is not rate-limited.
backend/routers/index.ts Fixed Show fixed Hide fixed
backend/routers/index.ts Fixed Show fixed Hide fixed
backend/routers/index.ts Fixed Show fixed Hide fixed
} catch (err) {
log.error('Failed to upload file', bbFile.filename);
console.error(err);
return res.status(500).send(err);

Check warning

Code scanning / CodeQL

Exception text reinterpreted as HTML Medium

Exception text
is reinterpreted as HTML without escaping meta-characters.
} catch (err) {
log.error('Failed to upload file', bbFile.filename);
console.error(err);
return res.status(500).send(err);

Check warning

Code scanning / CodeQL

Information exposure through a stack trace Medium

This information exposed to the user depends on
stack trace information
.
This information exposed to the user depends on
stack trace information
.

// Static routes
router.get('/', (req, res) => UserConfig.ready ? res.redirect('/') : res.render('setup'));
router.get('/ui.js', (req, res) => UserConfig.ready ? res.send('') : res.type('text/javascript').sendFile(path.join('dist-frontend/setup.mjs')));

Check failure

Code scanning / CodeQL

Missing rate limiting High

This route handler performs
a file system access
, but is not rate-limited.
backend/UserConfig.ts Fixed Show fixed Hide fixed
backend/UserConfig.ts Fixed Show fixed Hide fixed
backend/UserConfig.ts Fixed Show fixed Hide fixed
const Checkers: UserConfigTypeChecker = {
uploadsDir: (val) => {
try {
fs.pathExistsSync(val)

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.
uploadsDir: (val) => {
try {
fs.pathExistsSync(val)
? fs.accessSync(val)

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.
try {
fs.pathExistsSync(val)
? fs.accessSync(val)
: fs.mkdirSync(val, { recursive: true });

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.
Comment on lines 97 to 128
router.get('/direct/:fakeId', async (req, res) => {
if (!UserConfig.ready) res.redirect('/setup');

// Get the ID
const fakeId = req.params.fakeId;

if (!files.has(fakeId)) return res.status(404).send();
else {

// Get file metadata
const meta = files.get(fakeId)!;

// File data can come from either S3 or local filesystem
let output: Readable | NodeJS.ReadableStream;

// Try to retrieve the file
if (!!meta.save.s3) {
const file = await getFileS3(meta.fileKey);
if (!file.Body) return res.status(500).send('Unknown error');
output = file.Body as Readable;
} else output = fs.createReadStream(meta.save.local!);

// Configure response headers
res.type(meta.mimetype)
.header('Content-Disposition', `inline; filename="${meta.filename}"`)
.header('Cache-Control', 'public, max-age=31536000, immutable')
.header('Accept-Ranges', 'bytes');

// Send the file (thanks to https://stackoverflow.com/a/67373050)
output.pipe(res);
}
});

Check failure

Code scanning / CodeQL

Missing rate limiting High

This route handler performs
a file system access
, but is not rate-limited.
This route handler performs
a file system access
, but is not rate-limited.
return new Promise(async (resolve, reject) => {
try {
// Run query
const [rowz, _fields] = await MySql._pool.query(`SELECT Data FROM ${table} WHERE NanoID = '${key}';`);

Check failure

Code scanning / CodeQL

Database query built from user-controlled sources High

This query string depends on a
user-provided value
.
try {

// Get the file size
const size = (await fs.stat(bbFile.file)).size;

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.
const size = (await fs.stat(bbFile.file)).size;

// Get the hash
const sha256 = crypto.createHash('sha256').update(await fs.readFile(bbFile.file)).digest('base64');

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.

// Username check
if (!newUser.username) issue = 'Missing username'
newUser.username.replaceAll(/[^A-z0-9_-]/g, '');

Check warning

Code scanning / CodeQL

Overly permissive regular expression range Medium

Suspicious character range that is equivalent to [A-Z[]^_`a-z].
const sha256 = crypto.createHash('sha256').update(await fs.readFile(bbFile.file)).digest('base64');

// * Move the file
if (!s3) await fs.move(bbFile.file, destination);

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.
const sha256 = crypto.createHash('sha256').update(await fs.readFile(bbFile.file)).digest('base64');

// * Move the file
if (!s3) await fs.move(bbFile.file, destination);

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.

// * Move the file
if (!s3) await fs.move(bbFile.file, destination);
else await uploadFileS3(await fs.readFile(bbFile.file), fileKey, bbFile.mimetype, size, sha256);

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression High

This path depends on a
user-provided value
.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants