core/polyfills: Improved fetch polyfills (#234)

docs/api-reference/core/fetch-file.md

@@ -6,52 +6,62 @@ The `fetchFile` function is a wrapper around `fetch` which provides support for
 
 Use the `fetchFile` function as follows:
 
+```js
+import {fetchFile} from '@loaders.gl/core';
+
+const response = await fetchFile(url);
+
+// Now use standard browser Response APIs
+
+// Note: headers are case-insensitive
+const contentLength = response.headers.get('content-length');
+const mimeType = response.headers.get('content-type');
+
+const arrayBuffer = await response.arrayBuffer();
+```
+
+The `Response` object from `fetchFile` is usually passed to `parse` as follows:
+
 ```js
 import {fetchFile, parse} from '@loaders.gl/core';
 import {OBJLoader} from '@loaders.gl/obj';
 
-data = await parse(fetchFile(url), OBJLoader);
-// Application code here
-...
+const data = await parse(fetchFile(url), OBJLoader);
 ```
 
-If you don't care about the extra features in `fetchFile` just use the browders built-in `fetch` method.
+Note that if you don't need the extra features in `fetchFile`, you can just use the browsers built-in `fetch` method.
 
 ```js
 import {parse} from '@loaders.gl/core';
 import {OBJLoader} from '@loaders.gl/obj';
 
-data = await parse(fetch(url), OBJLoader);
-// Application code here
-...
+const data = await parse(fetch(url), OBJLoader);
 ```
 
 ## Functions
 
 ### fetchFile(url : String [, options : Object]) : Promise.Response
 
-A version of [`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Response) that:
+A wrapper around the platform [`fetch`](https://developer.mozilla.org/en-US/docs/Web/API/fetch) function with some additions:
 
 - Supports `setPathPrefix`: If path prefix has been set, it will be appended if `url` is relative (e.g. does not start with a `/`).
+- Supports `File` and `Blob` objects on the browser (and returns "mock" fetch response objects).
 
 Returns:
 
-- A promise that resolves into a fetch `Response` object, with the following methods/fields:
-  - `arrayBuffer() : Promise.ArrayBuffer` - Loads the file as an `ArrayBuffer`.
-  - `text() : Promise.String` - Loads the file and decodes it into text.
-  - `body : ReadableStream` - A stream that can be used to incrementally read the contents of the file.
+- A promise that resolves into a fetch [`Response`](https://developer.mozilla.org/en-US/docs/Web/API/Response) object, with the following methods/fields:
+    - `headers`: `Headers` - A [`Headers`](https://developer.mozilla.org/en-US/docs/Web/API/Headers) object.
+    - `arrayBuffer()`: Promise.ArrayBuffer` - Loads the file as an `ArrayBuffer`.
+    - `text()`: Promise.String` - Loads the file and decodes it into text.
+    - `json()`: Promise.String` - Loads the file and decodes it into JSON.
+    - `body` : ReadableStream` - A stream that can be used to incrementally read the contents of the file.
 
 Options:
 
 Under Node.js, options include (see [fs.createReadStream](https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options)):
 
 - `options.highWaterMark` (Number) Default: 64K (64 \* 1024) - Determines the "chunk size" of data read from the file.
 
-Remarks:
-
-- In the browser `fetchFile` will delegate to fetch after resolving the URL.
-- In node.js a mock `Response` object will be returned.
-
 ### readFileSync(url : String [, options : Object]) : ArrayBuffer | String
 
 > This function only works on Node.js or using data URLs.
@@ -64,6 +74,9 @@ Notes:
 
 ## Remarks
 
+- `fetchFile` will delegate to `fetch` after resolving the URL.
+- For some data sources such as node.js and `File`/`Blob` objects a mock `Response` object will be returned, and not all fields/members may be implemented.
+- When possible, `Content-Length` and `Content-Type` `headers` are also populated for non-request data sources including `File`, `Blob` and Node.js files.
 - `fetchFile` is intended to be a small (in terms of bundle size) function to help applications work with files in a portable way. The `Response` object returned on Node.js does not implement all the functionality the browser does. If you run into the need
 - In fact, the use of any of the file utilities including `readFile` and `readFileAsync` functions with other loaders.gl functions is entirely optional. loader objects can be used with data loaded via any mechanism the application prefers, e.g. directly using `fetch`, `XMLHttpRequest` etc.
 - The "path prefix" support is intentended to be a simple mechanism to support certain work-arounds. It is intended to help e.g. in situations like getting test cases to load data from the right place, but was never intended to support general application use cases.

docs/api-reference/core/load.md

@@ -10,17 +10,22 @@ The `load` function is used to load and parse data with a specific _loader objec
 
 The `loaders` parameter can also be omitted, in which case any _loader objects_ previously registered with [`registerLoaders`](docs/api-reference/core/register-loaders) will be used.
 
-- `url` - Can be a string, either a data url or a request url, or in Node.js, a file name, or in the browser, a File object. Or any format that could be accepted by [`parse`](https://github.com/uber-web/loaders.gl/blob/master/docs/api-reference/core/parse.md#parsedata--arraybuffer--string--options--object--url--string--promise). If `url` is not a `string`, will call `parse` directly.
-- `data` - loaded data, either in binary or text format.
+- `url` - Urls can be data urls (`data://`) or a request (`http://` or `https://`) urls, or a file name (Node.js only). Also accepts `File` or `Blob` object (Browser only). Can also accept any format that is accepted by [`parse`](https://github.com/uber-web/loaders.gl/blob/master/docs/api-reference/core/parse.md), with the exception of strings that are interpreted as urls.
 - `loaders` - can be a single loader or an array of loaders. If ommitted, will use the list of registered loaders (see `registerLoaders`)
 - `options` - optional, contains both options for the read process and options for the loader (see documentation of the specific loader).
-- `options.dataType`=`arraybuffer` - By default reads as binary. Set to 'text' to read as text.
+
+- `options.dataType`=`arraybuffer` - Default depends on loader object. Set to 'text' to read as text.
+
+`url` values
 
 Returns:
 
-- Return value depends on the _loader object_ category
+- Return value depends on the _loader category_.
 
 Notes:
 
+- If `url` is not a `string`, `load` will call `parse` directly.
 - Any path prefix set by `setPathPrefix` will be appended to relative urls.
 - `load` takes a `url` and a loader object, checks what type of data that loader prefers to work on (e.g. text, binary, stream, ...), loads the data in the appropriate way, and passes it to the loader.
+- If `@loaders.gl/polyfills` is installed, `load` will work under Node.js as well.
+

docs/api-reference/polyfills/README.md

@@ -34,15 +34,15 @@ The Node.js `fetch` polyfill supports a subset of the browser fetch API, includi
 
 `TextEncoder` and `TextDecoder` polyfills are provided to ensure these APIs are always available. In modern browsers these will evaluate to the built-in objects of the same name, however under Node.js polyfills are transparently installed.
 
-Note: The polyfill only guarantees UTF8 support.
+Note: The provided polyfills only guarantee UTF8 support.
 
 ## Remarks
 
+- Applications should only install this module if they need to run under older environments. While the polyfills are only installed at runtime if the platform does not already support them, importing this module will increase the application's bundle size.
 - Refer to browser documentation for the usage of these classes, e.g. MDN.
-- In the browser, overhead of using these imports is very low, as they refer to built-in classes.
-- If working under older browsers, e.g. IE11, you may need to install your own TextEncoder/TextDecoder polyfills before loading this library (to be confirmed...)
+- In the browser, overhead of using these imports is very low, as most polyfills are only bundled under Node.js.
+- If working under older browsers, e.g. IE11, you may need to install your own TextEncoder/TextDecoder polyfills before loading this library
 
+## Attribution
 
-## Remarks
-
-Note: Applications should only install this module if they need to run under older environments. While the polyfills are only installed at runtime if the platform does not already support them, importing this module will increase the application's bundle size.
+The `Header` polyfill (for Node.js `fetch`) is a fork of the implementation in https://github.com/github/fetch (MIT license).

modules/core/src/javascript-utils/is-type.js

@@ -1,4 +1,4 @@
-/* global File */
+/* global File, Blob */
 
 const isBoolean = x => typeof x === 'boolean';
 const isFunction = x => typeof x === 'function';
@@ -14,9 +14,11 @@ export const isIterator = x => isObject(x) && 'done' in x && 'value' in x;
 
 export const isFetchResponse = x =>
   (typeof window !== 'undefined' && x instanceof window.Response) ||
-  (x.arrayBuffer && x.json && x.body);
+  (x.arrayBuffer && x.text && x.json);
 
 export const isFile = x => typeof File !== 'undefined' && x instanceof File;
+export const isBlob = x => typeof Blob !== 'undefined' && x instanceof Blob;
+export const isFileReadable = x => isFile(x) || isBlob(x); // Blob & File are FileReader compatible
 
 export const isWritableDOMStream = x => {
   return isObject(x) && isFunction(x.abort) && isFunction(x.getWriter);

modules/core/src/lib/fetch/fetch-file.browser.js

@@ -1,56 +1,74 @@
-/* global FileReader */
+/* global FileReader, Headers */
 import assert from '../../utils/assert';
 
 // File reader fetch "polyfill" for the browser
-class FileResponse {
-  constructor(file) {
-    this._file = file;
-    this._promise = new Promise((resolve, reject) => {
-      try {
-        this._reader = new FileReader();
-        this._reader.onerror = error => reject(new Error(error));
-        this._reader.onabort = () => reject(new Error('Read aborted.'));
-        this._reader.onload = () => resolve(this._reader.result);
-      } catch (error) {
-        reject(error);
-      }
-    });
+class FileReadableResponse {
+  constructor(fileOrBlob) {
+    this._fileOrBlob = fileOrBlob;
+    this.bodyUsed = false;
   }
 
-  headers() {
-    return {};
+  get headers() {
+    return new Headers({
+      'Content-Length': this._fileOrBlob.size,
+      'Content-Type': this._fileOrBlob.type
+    });
   }
 
   url() {
-    return this._file.name;
+    // Note: This is just the file name without path information
+    // Note: File has `name` field but the Blob baseclass does not
+    return this._fileOrBlob.name || '';
   }
 
   async arrayBuffer() {
-    this.bodyUsed = true;
-    this._reader.readAsArrayBuffer(this._file);
-    return this._promise;
+    const {reader, promise} = this._getFileReader();
+    reader.readAsArrayBuffer(this._fileOrBlob);
+    return promise;
   }
 
   async text() {
-    this.bodyUsed = true;
-    this._reader.readAsText(this._file);
-    return this._promise;
+    const {reader, promise} = this._getFileReader();
+    reader.readAsText(this._fileOrBlob);
+    return promise;
   }
 
   async json() {
-    return JSON.parse(this.text());
+    const text = await this.text();
+    return JSON.parse(text);
   }
 
-  get body() {
-    // TODO - body, how to support stream?
-    // eslint-disable-next-line
-    // https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams#Creating_your_own_custom_readable_stream
-    assert(false);
+  // TODO - body, how to support stream?
+  // Can this be portable?
+  // eslint-disable-next-line
+  // https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams#Creating_your_own_custom_readable_stream
+  // get body() {
+  //   assert(false);
+  // }
+
+  // PRIVATE
+
+  _getFileReader() {
+    assert(!this.bodyUsed);
+    this.bodyUsed = true;
+
+    let reader;
+    const promise = new Promise((resolve, reject) => {
+      try {
+        reader = new FileReader();
+        reader.onerror = error => reject(new Error(error));
+        reader.onabort = () => reject(new Error('Read aborted.'));
+        reader.onload = () => resolve(reader.result);
+      } catch (error) {
+        reject(error);
+      }
+    });
+    return {reader, promise};
   }
 }
 
 // @param {File|Blob} file  HTML File or Blob object to read as string
 // @returns {Promise.string}  Resolves to a string containing file contents
-export default function fetchFileObject(file, options) {
-  return new FileResponse(file, options);
+export default function fetchFileReadable(fileOrBlob, options) {
+  return Promise.resolve(new FileReadableResponse(fileOrBlob, options));
 }

modules/core/src/lib/fetch/fetch-file.js

@@ -1,16 +1,16 @@
 /* global fetch */
-import {isFile} from '../../javascript-utils/is-type';
+import {isFileReadable} from '../../javascript-utils/is-type';
 import {resolvePath} from './file-aliases';
-import {readFileObject} from './read-file.browser';
+import fetchFileReadable from './fetch-file.browser';
 
 // As fetch but respects pathPrefix and file aliases
 // Reads file data from:
 // * data urls
 // * http/http urls
 // * File/Blob objects
 export async function fetchFile(url, options) {
-  if (isFile(url)) {
-    return readFileObject(url, options);
+  if (isFileReadable(url)) {
+    return fetchFileReadable(url, options);
   }
   url = resolvePath(url);
   // TODO - SUPPORT reading from `File` objects

modules/core/src/lib/fetch/read-file.browser.js

@@ -1,8 +1,5 @@
 // TODO - this file is not tested
-/* global fetch */
 import assert from '../../utils/assert';
-import {isFile} from '../../javascript-utils/is-type';
-import fetchFileObject from './fetch-file.browser';
 
 const DEFAULT_OPTIONS = {
   dataType: 'arraybuffer',
@@ -12,11 +9,6 @@ const DEFAULT_OPTIONS = {
 
 const isDataURL = url => url.startsWith('data:');
 
-export async function readFileObject(file, options) {
-  const response = fetchFileObject(file, options);
-  return options.dataType === 'text' ? response.text() : response.arrayBuffer();
-}
-
 // In a few cases (data URIs, files under Node) "files" can be read synchronously
 export function readFileSyncBrowser(uri, options) {
   options = getReadFileOptions(options);
@@ -41,30 +33,3 @@ function getReadFileOptions(options = {}) {
   options.responseType = options.responseType || options.dataType;
   return options;
 }
-
-// DEPRECATED
-
-// Reads raw file data from:
-// * http/http urls
-// * data urls
-// * File/Blob objects
-// etc?
-export async function readFile(uri, options = {}) {
-  options = getReadFileOptions(options);
-
-  // NOTE: data URLs are decoded by fetch
-
-  // SUPPORT reading from `File` objects
-  if (isFile(uri)) {
-    readFileObject(uri, options);
-  }
-
-  // In a web worker, XMLHttpRequest throws invalid URL error if using relative path
-  // resolve url relative to original base
-  // TODO - merge this into `resolvePath?
-  // uri = new URL(uri, location.href).href;
-
-  // Browser: Try to load all URLS via fetch, as they can be local requests (e.g. to a dev server)
-  const response = await fetch(uri, options);
-  return response[options.dataType]();
-}

modules/core/src/lib/load.js

@@ -1,4 +1,4 @@
-import {isFile} from '../javascript-utils/is-type';
+import {isFileReadable} from '../javascript-utils/is-type';
 import {fetchFile} from './fetch/fetch-file';
 import {isLoaderObject} from './loader-utils/normalize-loader';
 import {mergeLoaderAndUserOptions} from './loader-utils/normalize-options';
@@ -24,7 +24,7 @@ export async function load(url, loaders, options) {
   }
 
   // Extract a url for auto detection
-  const autoUrl = isFile(url) ? url.name : url;
+  const autoUrl = isFileReadable(url) ? url.name : url;
 
   loaders = loaders || getRegisteredLoaders();
   const loader = Array.isArray(loaders) ? autoDetectLoader(autoUrl, null, loaders) : loaders;
@@ -38,7 +38,7 @@ export async function load(url, loaders, options) {
 
   // at this point, data can be binary or text
   let data = url;
-  if (isFile(data) || typeof data === 'string') {
+  if (isFileReadable(data) || typeof data === 'string') {
     data = await fetchFile(url, options);
   }
   return parse(data, loaders, options, autoUrl);