Change proposal: break the fixed sample limit

[Borewit]

Limitation

Some file types cannot be determined reliable within the current 4k (or any reasonable single buffer limit). For example, an audio file with 600 kB ID3v2 header, may not be necessary an MP3 file. There are other audio formats which may be prefixed with an ID3v2 header.

Proposed solution

Using a stream based solution, we could read information, starting from the beginning required to determine file type. This is an approach I use music-metadata using strtok3 tokenizer, which supports peeking. It picks the most efficient way to jump to next required offset, depending if the underlying data is random accessible, like a file, blob or buffer. Or just read ahead in case of stream. This access abstraction can re-used from readable-web-to-node-stream.

what happens if a file type can be determined by the first 4 bytes?

The first 4 bytes will be read, no more.

what happens if the file is prefixed with ID3v2 header of 4 MB?

It will keep requesting data at the calculated offsets, in sequential manner, until it can determine the file type. But no more then required.

As this is major change, and will introduce a number of specialized function, e.g.:

// Determine file type from a node.js file
const type1 = fileType.fromFile(file);

// Determine file type from a node.js stream
const type2 = fileType.fromStream(stream);

// Determine file type using a buffer as an input (where the buffer represents the entire file)
const type3 = fileType.fromBuffer(buffer)` 

A dedicated module for browser usage (using is effectively an adapter) offers browser specific functions:

// Determine file type from a browser File or Blob
const type1 = browserFileType.fromBlob(blob);

// Determine file type from a browser ReadableStream
const type2 = browserFileType.fromStream(stream);

The browser part adapter is comparable with how I wrapped music-metadata with music-metadata-browser.

I love to hear your feedback to make an assessment if this proposal is worth implementing.

Looking forward to hear your feedback.

Related issues: #210

Implementation status

Development branch: stream-tokenizer

Progress / ToDo

• PR Initial integration with strtok3 #261:
  • Update AP
  • Proof backward compatibility, being able to determine file type from a partial file as a buffer
• PR Use tokenizer #262:
  • Utilize tokenizer (migrate checks to use tokenizer)
  • Update TypeScript definitions
• PR Separate Node.js dependency #265
  • Separate core and Node.js functions by sub-inclusion
• PR Change proposal: break the fixed sample limit #248
  • Add /browser sub-module

[sindresorhus]

This sounds reasonable. I think it should still be possible to pass a partial file as a buffer for the detections that doesn't need the whole file.

[Borewit]

I think it should still be possible to pass a partial file as a buffer for the detections that doesn't need the whole file.

Fair enough.

[Borewit]

@sindresorhus can you please open a 'development' branch, e.g. stream-tokenizer, allowing to divide this effort over multiple PR's against that branch?

[sindresorhus]

https://github.com/sindresorhus/file-type/tree/stream-tokenizer

[Borewit]

@sindresorhus, implementation is ready for review.