Skip to content

[Components] Support file URLS and file paths in a single prop input #16977

Open
@js07

Description

@js07

Summary

Update components that accept a file path or file URL to accept both via the same prop, leveraging the getFileStream and getFileStreamAndMetadata functions from @pipedream/platform.

Background

The @pipedream/platform package now includes utility functions that can handle both local file paths and remote URLs:

  • getFileStream(pathOrUrl) - Returns a readable stream from either a local file or remote URL
  • getFileStreamAndMetadata(pathOrUrl) - Returns both stream and metadata (size, content type, etc.)
  • Metadata extraction from HTTP headers (content-length, last-modified, etag, content-type)
  • Fallback to temporary file download when content-length is unavailable

Proposed Changes

  1. Update component props: Consolidate filePath and fileUrl props to a single file prop with a description to indicate support for both file paths and URLs
  2. Replace direct file operations: Update components to use getFileStream() or getFileStreamAndMetadata() instead of direct fs operations

Benefits

  • Unified interface: Users can provide either local files or remote URLs without needing separate props
  • Consistent behavior: All file-handling components work the same way
  • Better UX: Users can directly reference files from cloud storage, APIs, or other HTTP sources
  • Leverages existing code: No need to reimplement URL fetching logic

Example Usage

// Current: only file paths
import fs from "fs";
export default {
  props: {
    filePath: {
      type: "string",
      description: "The path to the file saved to the `/tmp` directory (e.g. `/tmp/example.pdf`)"
    }
  },
  async run({ steps, $ }) {
    const stream = fs.createReadStream(this.filePath);
    // ...
  }
};

// Enhanced: file paths AND URLs
import { getFileStream } from "@pipedream/platform";
export default {
  props: {
    file: {
      type: "string",
      label: "File",
      description: "Provide either a file URL or a path to a file in the /tmp directory (for example, /tmp/myFlie.pdf).",
    }
  },
  async run({ steps, $ }) {
    const stream = await getFileStream(this.file);
    // Works with both "/tmp/file.txt" and "https://example.com/file.txt"
  }
};

// For use with `form-data`
import FormData from "form-data";
import { getFileStreamAndMetadata } from "@pipedream/platform";
export default {
  props: {
    file: {
      type: "string",
      label: "File",
      description: "Provide either a file URL or a path to a file in the /tmp directory (for example, /tmp/myFlie.pdf).",
    }
  },
  async run({ steps, $ }) {
    const { stream, metadata } = await getFileStreamAndMetadata(this.file);
    // Works with both "/tmp/file.txt" and "https://example.com/file.txt"
    const form = new FormData();
    const {
      stream, metadata,
    } = await getFileStreamAndMetadata(file);
    form.append("file", stream, {
      contentType: metadata.contentType,
      knownLength: metadata.size,
      filename: metadata.name,
    });
  }
};

Implementation Checklist

  • Audit existing components with filePath/file props
  • Update components to use getFileStream/getFileStreamAndMetadata
  • Update props and prop descriptions

Metadata

Metadata

Labels

Type

No type

Projects

Status

Doing

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions