Open
Description
Summary
Update components that accept a file path or file URL to accept both via the same prop, leveraging the getFileStream
and getFileStreamAndMetadata
functions from @pipedream/platform
.
Background
The @pipedream/platform
package now includes utility functions that can handle both local file paths and remote URLs:
getFileStream(pathOrUrl)
- Returns a readable stream from either a local file or remote URLgetFileStreamAndMetadata(pathOrUrl)
- Returns both stream and metadata (size, content type, etc.)- Metadata extraction from HTTP headers (content-length, last-modified, etag, content-type)
- Fallback to temporary file download when content-length is unavailable
Proposed Changes
- Update component props: Consolidate
filePath
andfileUrl
props to a singlefile
prop with a description to indicate support for both file paths and URLs - Replace direct file operations: Update components to use
getFileStream()
orgetFileStreamAndMetadata()
instead of directfs
operations
Benefits
- Unified interface: Users can provide either local files or remote URLs without needing separate props
- Consistent behavior: All file-handling components work the same way
- Better UX: Users can directly reference files from cloud storage, APIs, or other HTTP sources
- Leverages existing code: No need to reimplement URL fetching logic
Example Usage
// Current: only file paths
import fs from "fs";
export default {
props: {
filePath: {
type: "string",
description: "The path to the file saved to the `/tmp` directory (e.g. `/tmp/example.pdf`)"
}
},
async run({ steps, $ }) {
const stream = fs.createReadStream(this.filePath);
// ...
}
};
// Enhanced: file paths AND URLs
import { getFileStream } from "@pipedream/platform";
export default {
props: {
file: {
type: "string",
label: "File",
description: "Provide either a file URL or a path to a file in the /tmp directory (for example, /tmp/myFlie.pdf).",
}
},
async run({ steps, $ }) {
const stream = await getFileStream(this.file);
// Works with both "/tmp/file.txt" and "https://example.com/file.txt"
}
};
// For use with `form-data`
import FormData from "form-data";
import { getFileStreamAndMetadata } from "@pipedream/platform";
export default {
props: {
file: {
type: "string",
label: "File",
description: "Provide either a file URL or a path to a file in the /tmp directory (for example, /tmp/myFlie.pdf).",
}
},
async run({ steps, $ }) {
const { stream, metadata } = await getFileStreamAndMetadata(this.file);
// Works with both "/tmp/file.txt" and "https://example.com/file.txt"
const form = new FormData();
const {
stream, metadata,
} = await getFileStreamAndMetadata(file);
form.append("file", stream, {
contentType: metadata.contentType,
knownLength: metadata.size,
filename: metadata.name,
});
}
};
Implementation Checklist
- Audit existing components with
filePath
/file
props - Update components to use
getFileStream
/getFileStreamAndMetadata
- Update props and prop descriptions
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Doing