Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

make buffer[index] work #14

Merged
merged 19 commits into from

5 participants

@substack

This unfortunately took a lot of commits since buffer[index] only works with the underlying SlowBuffer and not with Buffer, so I had to merge all the method signatures from Buffer over top of the SlowBuffer implementations.

Despite the sheer volume of changes, all the existing tests pass plus some new ones. After I got all the tests to pass in node, I inlined all the comparisons with the built-in Buffer so that we can run the test suite in browsers.

I also included a "testling" field for the package.json so we can verify which browsers are currently working and which ones are failing.
Here's the status for my fork: https://ci.testling.com/substack/buffer-browserify

To add a hook for testling, add http://git.testling.com to the "WebHook URLs" section of the server hooks page. Once the hook is working you can add a badge to the readme with:

[![browser compatibility](https://ci.testling.com/toots/buffer-browserify.png)](https://ci.testling.com/toots/buffer-browserify)
@substack substack referenced this pull request
Open

node.js simple tests #1

@toots
Owner

Thank you so much! I will review and merge this very soon!!

@substack

We're having some issues with testling-ci right now but I ran the test suite locally in chrome and it was completely passing.

@tonistiigi

Just wanted to make a comment that by adding this we (IMHO) are removing the possibility to ever support typed arrays as a backing for the buffers. I think there is no way to subclass Uint8Array. (In browsers that support __proto__ it could be done for read only mode).

The issue I'm having:

  • User selects file with the input field
  • I want to use some node module on the data or just upload with request so I convert Blob to Buffer(very slow for loop copy to dynamic array)
  • I upload it with request. In http-browserify internals the buffer is again converted from dynamic array to a typed array.
  • So I get 2 slow copies(freezes browser even on small files) when actually no copies are needed at all because all of them could use same ArrayBuffer as a backing.

I was planning to rewrite current SlowBuffer implementation so it would switch to Uint8Array when possible and not make copies when not needed.

Or are there better alternatives to make buffer-browserify play nice with the binary formats already present in the browser? I realize that having index accessors it also important.

@substack I don't get the slicing? It just sets the offset/length property but doesn't even keep the reference to the subject(parent)?

@substack

@tonistiigi The problem with re-encoding is present in node when people use typed arrays too. buffer-browserify is just node's Buffer API which does wrap around like a Uint8Array would do. That said, I think it would be much easier to update the patched code here to start using Uint8Array than to use the original code since in the original code you'll need to juggle Buffer and SlowBuffer api surface area instead of just Buffer.

@tonistiigi

@substack

The difference of course is that in Node you can use Buffer for all your binary needs but in the browser there are many APIs that know nothing about buffers and use typed arrays instead.

I though about it a bit more and think my problem should have a better solution by just adding 2 new methods Buffer.fromUint8Array() and Buffer.prototype.toUint8Array() (or bufferToUint8Array(b) etc). And I think they should be included with browserify even though they don't exists in Node because things like http-browserify should use them.

The slicing issue still remains. Currently it doesn't work at all like in Node.

@substack

A separate module could handle the {from,to}Uint8Array logic since that would work in both node and browsers. This patch doesn't keep a reference to the parent because that doesn't work with indexing. I don't think I've ever seen any code that depends on the buffer.parent internals except for the buffer implementation itself.

@tonistiigi

I'm not taking specifically about the need have a reference to parent. With this patch when you call slice() for the buffer the returning buffer does not have any data associated with it. You can't use the index properties or read* methods for that that buffer.

@toots
Owner

Hi,

Just read the PR diff and tried the tests. It's unfortunate that there is so much diff but it seems to be working fine.

Concerning the issue with typed arrays and html-specific javascript APIs for binary blobs, I agree with @substack , it's not something that can or should be tackled here. Having a Buffer API that works is already good enough.

Mixing with HTML5 APIs is another beast that'd require much more though and a bigger project. It could go either way, i.e. having a node implementation of HTML5 file and blob API or having an optional Buffer implementation using HTML5 API in buffer-browserify .. Makes my head spin :-)

I'll have another pass at the PR and merge it later tonight or tomorrow if all is still well..

@tonistiigi tonistiigi referenced this pull request from a commit in tonistiigi/buffer-browserify
@tonistiigi tonistiigi Add simple failing slice test to #14 2eb4d47
@juliangruber

just stumbled over this today, big +1 from me.

@toots toots merged commit b9492bf into toots:master
@tonistiigi

I don't really get why everyone is ignoring the fact that buffer.slice() doesn't work any more. This PR brakes every piece of code that I have ever written that uses buffer-browserify.

@toots
Owner

It's not being ignore on my side, just not implemented yet with the new buf[i]. buffer-browserify hasn't been released yet for this reason. PR welcome btw :)

@chrisdickinson

this actually worries me a lot, perf-wise. it changes the semantics of .slice from "no copy" to "full copy", which will dramatically affect a lot of the libs I've been writing.

@toots
Owner

Alright guys, I need to make a decision real quick because other people are starting to work on the master branch, even though it's not been released yet.

I personally believe that performance has never been an important choice because the goal of the module is to be as compatible as possible with node's original API, regardless of performances.

It seems unreasonable to me to write node code, wrap it for the browser and expect it to have competitive performances there... Low-level APIs, like file access, etc.. are too different in my opinion to grant this in general.

Thus, I'm tempted to move on with the current support for buf[i] add a slow slice and call it a day...

I'm open to be convinced otherwise, tho.

@tonistiigi

Its not only performance. In Node the memory is shared so if you change one the other one will also change. This can't be achieved with a copy.

@tonistiigi tonistiigi referenced this pull request from a commit in tonistiigi/buffer-browserify
@tonistiigi tonistiigi Add another slice test to #14 ae792a9
@toots
Owner

Hi,

Been reviewing the code wrt to previous behaviour of the slice function and it appears that slice was indeed working as in node before, by sharing data between the original Buffer and the sliced one.

I also think that it is not possible to implement this behaviour in the code after merging this PR. I'd like to be proven wrong tho.

Consequently, my conclusion is that the this[i] API is mutually exclusive with a proper slice implementation. As such, I think slice is kinda important and, regarding this[i], I'd rather have no API at all that than an API that either diverges from node's behavior or does not implement slice.

Thus, I am tempted to revert the merge of #14.

Let me know what you guys think..?

@substack

Why isn't this on npm?

@toots
Owner

@substack because I am not sure that this is a good idea. What is your thinking on the slice issue discussed above?

@toots
Owner

Ok. After much delay and some thinking, I'll release with these chances and no slice implemented (for now). Seems to be that the likelihood of programmers from node using buf[i] is much greater than the likelihood of programmers expecting changes in slice'd buffer to be reflected in original buffer. Will prolly add slice soon as well.

@tonistiigi

@toots I would suggest you to also merge my failing tests so people can see what currently works and what doesn't. Maybe solution is found and these start to pass. Or slice() should throw an error to let the user know that this is not something that can be used any more.

btw, regarding the speed issues I discussed before I have meanwhile written a module that solves that for me https://github.com/tonistiigi/uint8 . Too bad it will not work in this updated version.

@andris9 andris9 referenced this pull request in andris9/mimelib
Closed

Deprecation warning on node 0.11.6 #23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 16, 2013
  1. @substack
  2. @substack
Commits on Mar 20, 2013
  1. @substack
  2. @substack

    indexes test passes

    substack authored
  3. @substack
  4. @substack
  5. @substack
  6. @substack

    %s/SlowBuffer/Buffer/g

    substack authored
  7. @substack
  8. @substack

    port fill() to new buffer

    substack authored
  9. @substack

    ported .copy()

    substack authored
  10. @substack
  11. @substack

    fill with 0s for Buffer(num)

    substack authored
  12. @substack
  13. @substack
  14. @substack

    stub out SlowBuffer=Buffer and a phony poolSize to make this module l…

    substack authored
    …ook more like the core api
  15. @substack
  16. @substack

    using testling-ci

    substack authored
  17. @substack
This page is out of date. Refresh to see the latest.
Showing with 311 additions and 509 deletions.
  1. +209 −455 index.js
  2. +14 −2 package.json
  3. +59 −52 test/buffer.js
  4. +29 −0 test/indexes.js
View
664 index.js
@@ -1,45 +1,74 @@
-function SlowBuffer (size) {
- this.length = size;
-};
-
var assert = require('assert');
-
+exports.Buffer = Buffer;
+exports.SlowBuffer = Buffer;
+Buffer.poolSize = 8192;
exports.INSPECT_MAX_BYTES = 50;
+function Buffer(subject, encoding, offset) {
+ if (!(this instanceof Buffer)) {
+ return new Buffer(subject, encoding, offset);
+ }
+ this.parent = this;
+ this.offset = 0;
-function toHex(n) {
- if (n < 16) return '0' + n.toString(16);
- return n.toString(16);
-}
+ var type;
-function utf8ToBytes(str) {
- var byteArray = [];
- for (var i = 0; i < str.length; i++)
- if (str.charCodeAt(i) <= 0x7F)
- byteArray.push(str.charCodeAt(i));
- else {
- var h = encodeURIComponent(str.charAt(i)).substr(1).split('%');
- for (var j = 0; j < h.length; j++)
- byteArray.push(parseInt(h[j], 16));
- }
+ // Are we slicing?
+ if (typeof offset === 'number') {
+ this.length = coerce(encoding);
+ this.offset = offset;
+ } else {
+ // Find the length
+ switch (type = typeof subject) {
+ case 'number':
+ this.length = coerce(subject);
+ break;
- return byteArray;
-}
+ case 'string':
+ this.length = Buffer.byteLength(subject, encoding);
+ break;
-function asciiToBytes(str) {
- var byteArray = []
- for (var i = 0; i < str.length; i++ )
- // Node's code seems to be doing this and not & 0x7F..
- byteArray.push( str.charCodeAt(i) & 0xFF );
+ case 'object': // Assume object is an array
+ this.length = coerce(subject.length);
+ break;
- return byteArray;
-}
+ default:
+ throw new Error('First argument needs to be a number, ' +
+ 'array or string.');
+ }
-function base64ToBytes(str) {
- return require("base64-js").toByteArray(str);
+ // Treat array-ish objects as a byte array.
+ if (isArrayIsh(subject)) {
+ for (var i = 0; i < this.length; i++) {
+ if (subject instanceof Buffer) {
+ this[i] = subject.readUInt8(i);
+ }
+ else {
+ this[i] = subject[i];
+ }
+ }
+ } else if (type == 'string') {
+ // We are a string
+ this.length = this.write(subject, 0, encoding);
+ } else if (type === 'number') {
+ for (var i = 0; i < this.length; i++) {
+ this[i] = 0;
+ }
+ }
+ }
}
-SlowBuffer.byteLength = function (str, encoding) {
+Buffer.prototype.get = function get(i) {
+ if (i < 0 || i >= this.length) throw new Error('oob');
+ return this[i];
+};
+
+Buffer.prototype.set = function set(i, v) {
+ if (i < 0 || i >= this.length) throw new Error('oob');
+ return this[i] = v;
+};
+
+Buffer.byteLength = function (str, encoding) {
switch (encoding || "utf8") {
case 'hex':
return str.length / 2;
@@ -60,49 +89,29 @@ SlowBuffer.byteLength = function (str, encoding) {
}
};
-function blitBuffer(src, dst, offset, length) {
- var pos, i = 0;
- while (i < length) {
- if ((i+offset >= dst.length) || (i >= src.length))
- break;
-
- dst[i + offset] = src[i];
- i++;
- }
- return i;
-}
-
-SlowBuffer.prototype.utf8Write = function (string, offset, length) {
+Buffer.prototype.utf8Write = function (string, offset, length) {
var bytes, pos;
- return SlowBuffer._charsWritten = blitBuffer(utf8ToBytes(string), this, offset, length);
+ return Buffer._charsWritten = blitBuffer(utf8ToBytes(string), this, offset, length);
};
-SlowBuffer.prototype.asciiWrite = function (string, offset, length) {
+Buffer.prototype.asciiWrite = function (string, offset, length) {
var bytes, pos;
- return SlowBuffer._charsWritten = blitBuffer(asciiToBytes(string), this, offset, length);
+ return Buffer._charsWritten = blitBuffer(asciiToBytes(string), this, offset, length);
};
-SlowBuffer.prototype.binaryWrite = SlowBuffer.prototype.asciiWrite;
+Buffer.prototype.binaryWrite = Buffer.prototype.asciiWrite;
-SlowBuffer.prototype.base64Write = function (string, offset, length) {
+Buffer.prototype.base64Write = function (string, offset, length) {
var bytes, pos;
- return SlowBuffer._charsWritten = blitBuffer(base64ToBytes(string), this, offset, length);
+ return Buffer._charsWritten = blitBuffer(base64ToBytes(string), this, offset, length);
};
-SlowBuffer.prototype.base64Slice = function (start, end) {
+Buffer.prototype.base64Slice = function (start, end) {
var bytes = Array.prototype.slice.apply(this, arguments)
return require("base64-js").fromByteArray(bytes);
-}
-
-function decodeUtf8Char(str) {
- try {
- return decodeURIComponent(str);
- } catch (err) {
- return String.fromCharCode(0xFFFD); // UTF 8 invalid char
- }
-}
+};
-SlowBuffer.prototype.utf8Slice = function () {
+Buffer.prototype.utf8Slice = function () {
var bytes = Array.prototype.slice.apply(this, arguments);
var res = "";
var tmp = "";
@@ -120,7 +129,7 @@ SlowBuffer.prototype.utf8Slice = function () {
return res + decodeUtf8Char(tmp);
}
-SlowBuffer.prototype.asciiSlice = function () {
+Buffer.prototype.asciiSlice = function () {
var bytes = Array.prototype.slice.apply(this, arguments);
var ret = "";
for (var i = 0; i < bytes.length; i++)
@@ -128,9 +137,9 @@ SlowBuffer.prototype.asciiSlice = function () {
return ret;
}
-SlowBuffer.prototype.binarySlice = SlowBuffer.prototype.asciiSlice;
+Buffer.prototype.binarySlice = Buffer.prototype.asciiSlice;
-SlowBuffer.prototype.inspect = function() {
+Buffer.prototype.inspect = function() {
var out = [],
len = this.length;
for (var i = 0; i < len; i++) {
@@ -140,11 +149,11 @@ SlowBuffer.prototype.inspect = function() {
break;
}
}
- return '<SlowBuffer ' + out.join(' ') + '>';
+ return '<Buffer ' + out.join(' ') + '>';
};
-SlowBuffer.prototype.hexSlice = function(start, end) {
+Buffer.prototype.hexSlice = function(start, end) {
var len = this.length;
if (!start || start < 0) start = 0;
@@ -158,7 +167,7 @@ SlowBuffer.prototype.hexSlice = function(start, end) {
};
-SlowBuffer.prototype.toString = function(encoding, start, end) {
+Buffer.prototype.toString = function(encoding, start, end) {
encoding = String(encoding || 'utf8').toLowerCase();
start = +start || 0;
if (typeof end == 'undefined') end = this.length;
@@ -195,7 +204,7 @@ SlowBuffer.prototype.toString = function(encoding, start, end) {
};
-SlowBuffer.prototype.hexWrite = function(string, offset, length) {
+Buffer.prototype.hexWrite = function(string, offset, length) {
offset = +offset || 0;
var remaining = this.length - offset;
if (!length) {
@@ -220,12 +229,12 @@ SlowBuffer.prototype.hexWrite = function(string, offset, length) {
if (isNaN(byte)) throw new Error('Invalid hex string');
this[offset + i] = byte;
}
- SlowBuffer._charsWritten = i * 2;
+ Buffer._charsWritten = i * 2;
return i;
};
-SlowBuffer.prototype.write = function(string, offset, length, encoding) {
+Buffer.prototype.write = function(string, offset, length, encoding) {
// Support both (string, offset, length, encoding)
// and the legacy (string, encoding, offset, length)
if (isFinite(offset)) {
@@ -280,7 +289,7 @@ SlowBuffer.prototype.write = function(string, offset, length, encoding) {
// slice(start, end)
-SlowBuffer.prototype.slice = function(start, end) {
+Buffer.prototype.slice = function(start, end) {
if (end === undefined) end = this.length;
if (end > this.length) {
@@ -293,126 +302,86 @@ SlowBuffer.prototype.slice = function(start, end) {
return new Buffer(this, end - start, +start);
};
-SlowBuffer.prototype.copy = function(target, targetstart, sourcestart, sourceend) {
- var temp = [];
- for (var i=sourcestart; i<sourceend; i++) {
- assert.ok(typeof this[i] !== 'undefined', "copying undefined buffer bytes!");
- temp.push(this[i]);
- }
+// copy(targetBuffer, targetStart=0, sourceStart=0, sourceEnd=buffer.length)
+Buffer.prototype.copy = function(target, target_start, start, end) {
+ var source = this;
+ start || (start = 0);
+ end || (end = this.length);
+ target_start || (target_start = 0);
- for (var i=targetstart; i<targetstart+temp.length; i++) {
- target[i] = temp[i-targetstart];
- }
-};
+ if (end < start) throw new Error('sourceEnd < sourceStart');
-SlowBuffer.prototype.fill = function(value, start, end) {
- if (end > this.length) {
- throw new Error('oob');
- }
- if (start > end) {
- throw new Error('oob');
- }
+ // Copy 0 bytes; we're done
+ if (end === start) return 0;
+ if (target.length == 0 || source.length == 0) return 0;
- for (var i = start; i < end; i++) {
- this[i] = value;
+ if (target_start < 0 || target_start >= target.length) {
+ throw new Error('targetStart out of bounds');
}
-}
-function coerce(length) {
- // Coerce length to a number (possibly NaN), round up
- // in case it's fractional (e.g. 123.456) then do a
- // double negate to coerce a NaN to 0. Easy, right?
- length = ~~Math.ceil(+length);
- return length < 0 ? 0 : length;
-}
-
-
-// Buffer
-
-function Buffer(subject, encoding, offset) {
- if (!(this instanceof Buffer)) {
- return new Buffer(subject, encoding, offset);
+ if (start < 0 || start >= source.length) {
+ throw new Error('sourceStart out of bounds');
}
- var type;
+ if (end < 0 || end > source.length) {
+ throw new Error('sourceEnd out of bounds');
+ }
- // Are we slicing?
- if (typeof offset === 'number') {
- this.length = coerce(encoding);
- this.parent = subject;
- this.offset = offset;
- } else {
- // Find the length
- switch (type = typeof subject) {
- case 'number':
- this.length = coerce(subject);
- break;
+ // Are we oob?
+ if (end > this.length) {
+ end = this.length;
+ }
- case 'string':
- this.length = Buffer.byteLength(subject, encoding);
- break;
+ if (target.length - target_start < end - start) {
+ end = target.length - target_start + start;
+ }
- case 'object': // Assume object is an array
- this.length = coerce(subject.length);
- break;
+ var temp = [];
+ for (var i=start; i<end; i++) {
+ assert.ok(typeof this[i] !== 'undefined', "copying undefined buffer bytes!");
+ temp.push(this[i]);
+ }
- default:
- throw new Error('First argument needs to be a number, ' +
- 'array or string.');
- }
+ for (var i=target_start; i<target_start+temp.length; i++) {
+ target[i] = temp[i-target_start];
+ }
+};
- if (this.length > Buffer.poolSize) {
- // Big buffer, just alloc one.
- this.parent = new SlowBuffer(this.length);
- this.offset = 0;
-
- } else {
- // Small buffer.
- if (!pool || pool.length - pool.used < this.length) allocPool();
- this.parent = pool;
- this.offset = pool.used;
- pool.used += this.length;
- }
+// fill(value, start=0, end=buffer.length)
+Buffer.prototype.fill = function fill(value, start, end) {
+ value || (value = 0);
+ start || (start = 0);
+ end || (end = this.length);
- // Treat array-ish objects as a byte array.
- if (isArrayIsh(subject)) {
- for (var i = 0; i < this.length; i++) {
- if (subject instanceof Buffer) {
- this.parent[i + this.offset] = subject.readUInt8(i);
- }
- else {
- this.parent[i + this.offset] = subject[i];
- }
- }
- } else if (type == 'string') {
- // We are a string
- this.length = this.write(subject, 0, encoding);
- }
+ if (typeof value === 'string') {
+ value = value.charCodeAt(0);
+ }
+ if (!(typeof value === 'number') || isNaN(value)) {
+ throw new Error('value is not a number');
}
-}
+ if (end < start) throw new Error('end < start');
-function isArrayIsh(subject) {
- return Array.isArray(subject) || Buffer.isBuffer(subject) ||
- subject && typeof subject === 'object' &&
- typeof subject.length === 'number';
-}
+ // Fill 0 bytes; we're done
+ if (end === start) return 0;
+ if (this.length == 0) return 0;
-exports.SlowBuffer = SlowBuffer;
-exports.Buffer = Buffer;
+ if (start < 0 || start >= this.length) {
+ throw new Error('start out of bounds');
+ }
-Buffer.poolSize = 8 * 1024;
-var pool;
+ if (end < 0 || end > this.length) {
+ throw new Error('end out of bounds');
+ }
-function allocPool() {
- pool = new SlowBuffer(Buffer.poolSize);
- pool.used = 0;
+ for (var i = start; i < end; i++) {
+ this[i] = value;
+ }
}
-
// Static methods
Buffer.isBuffer = function isBuffer(b) {
- return b instanceof Buffer || b instanceof SlowBuffer;
+ return b instanceof Buffer || b instanceof Buffer;
};
Buffer.concat = function (list, totalLength) {
@@ -445,261 +414,75 @@ Buffer.concat = function (list, totalLength) {
return buffer;
};
-// Inspect
-Buffer.prototype.inspect = function inspect() {
- var out = [],
- len = this.length;
+// helpers
- for (var i = 0; i < len; i++) {
- out[i] = toHex(this.parent[i + this.offset]);
- if (i == exports.INSPECT_MAX_BYTES) {
- out[i + 1] = '...';
- break;
- }
- }
-
- return '<Buffer ' + out.join(' ') + '>';
-};
-
-
-Buffer.prototype.get = function get(i) {
- if (i < 0 || i >= this.length) throw new Error('oob');
- return this.parent[this.offset + i];
-};
-
-
-Buffer.prototype.set = function set(i, v) {
- if (i < 0 || i >= this.length) throw new Error('oob');
- return this.parent[this.offset + i] = v;
-};
+function coerce(length) {
+ // Coerce length to a number (possibly NaN), round up
+ // in case it's fractional (e.g. 123.456) then do a
+ // double negate to coerce a NaN to 0. Easy, right?
+ length = ~~Math.ceil(+length);
+ return length < 0 ? 0 : length;
+}
+function isArrayIsh(subject) {
+ return Array.isArray(subject) || Buffer.isBuffer(subject) ||
+ subject && typeof subject === 'object' &&
+ typeof subject.length === 'number';
+}
-// write(string, offset = 0, length = buffer.length-offset, encoding = 'utf8')
-Buffer.prototype.write = function(string, offset, length, encoding) {
- // Support both (string, offset, length, encoding)
- // and the legacy (string, encoding, offset, length)
- if (isFinite(offset)) {
- if (!isFinite(length)) {
- encoding = length;
- length = undefined;
- }
- } else { // legacy
- var swap = encoding;
- encoding = offset;
- offset = length;
- length = swap;
- }
+function toHex(n) {
+ if (n < 16) return '0' + n.toString(16);
+ return n.toString(16);
+}
- offset = +offset || 0;
- var remaining = this.length - offset;
- if (!length) {
- length = remaining;
- } else {
- length = +length;
- if (length > remaining) {
- length = remaining;
+function utf8ToBytes(str) {
+ var byteArray = [];
+ for (var i = 0; i < str.length; i++)
+ if (str.charCodeAt(i) <= 0x7F)
+ byteArray.push(str.charCodeAt(i));
+ else {
+ var h = encodeURIComponent(str.charAt(i)).substr(1).split('%');
+ for (var j = 0; j < h.length; j++)
+ byteArray.push(parseInt(h[j], 16));
}
- }
- encoding = String(encoding || 'utf8').toLowerCase();
- var ret;
- switch (encoding) {
- case 'hex':
- ret = this.parent.hexWrite(string, this.offset + offset, length);
- break;
-
- case 'utf8':
- case 'utf-8':
- ret = this.parent.utf8Write(string, this.offset + offset, length);
- break;
+ return byteArray;
+}
- case 'ascii':
- ret = this.parent.asciiWrite(string, this.offset + offset, length);
- break;
+function asciiToBytes(str) {
+ var byteArray = []
+ for (var i = 0; i < str.length; i++ )
+ // Node's code seems to be doing this and not & 0x7F..
+ byteArray.push( str.charCodeAt(i) & 0xFF );
- case 'binary':
- ret = this.parent.binaryWrite(string, this.offset + offset, length);
- break;
+ return byteArray;
+}
- case 'base64':
- // Warning: maxLength not taken into account in base64Write
- ret = this.parent.base64Write(string, this.offset + offset, length);
- break;
+function base64ToBytes(str) {
+ return require("base64-js").toByteArray(str);
+}
- case 'ucs2':
- case 'ucs-2':
- ret = this.parent.ucs2Write(string, this.offset + offset, length);
+function blitBuffer(src, dst, offset, length) {
+ var pos, i = 0;
+ while (i < length) {
+ if ((i+offset >= dst.length) || (i >= src.length))
break;
- default:
- throw new Error('Unknown encoding');
- }
-
- Buffer._charsWritten = SlowBuffer._charsWritten;
-
- return ret;
-};
-
-
-// toString(encoding, start=0, end=buffer.length)
-Buffer.prototype.toString = function(encoding, start, end) {
- encoding = String(encoding || 'utf8').toLowerCase();
-
- if (typeof start == 'undefined' || start < 0) {
- start = 0;
- } else if (start > this.length) {
- start = this.length;
- }
-
- if (typeof end == 'undefined' || end > this.length) {
- end = this.length;
- } else if (end < 0) {
- end = 0;
- }
-
- start = start + this.offset;
- end = end + this.offset;
-
- switch (encoding) {
- case 'hex':
- return this.parent.hexSlice(start, end);
-
- case 'utf8':
- case 'utf-8':
- return this.parent.utf8Slice(start, end);
-
- case 'ascii':
- return this.parent.asciiSlice(start, end);
-
- case 'binary':
- return this.parent.binarySlice(start, end);
-
- case 'base64':
- return this.parent.base64Slice(start, end);
-
- case 'ucs2':
- case 'ucs-2':
- return this.parent.ucs2Slice(start, end);
-
- default:
- throw new Error('Unknown encoding');
- }
-};
-
-
-// byteLength
-Buffer.byteLength = SlowBuffer.byteLength;
-
-
-// fill(value, start=0, end=buffer.length)
-Buffer.prototype.fill = function fill(value, start, end) {
- value || (value = 0);
- start || (start = 0);
- end || (end = this.length);
-
- if (typeof value === 'string') {
- value = value.charCodeAt(0);
- }
- if (!(typeof value === 'number') || isNaN(value)) {
- throw new Error('value is not a number');
- }
-
- if (end < start) throw new Error('end < start');
-
- // Fill 0 bytes; we're done
- if (end === start) return 0;
- if (this.length == 0) return 0;
-
- if (start < 0 || start >= this.length) {
- throw new Error('start out of bounds');
- }
-
- if (end < 0 || end > this.length) {
- throw new Error('end out of bounds');
- }
-
- return this.parent.fill(value,
- start + this.offset,
- end + this.offset);
-};
-
-
-// copy(targetBuffer, targetStart=0, sourceStart=0, sourceEnd=buffer.length)
-Buffer.prototype.copy = function(target, target_start, start, end) {
- var source = this;
- start || (start = 0);
- end || (end = this.length);
- target_start || (target_start = 0);
-
- if (end < start) throw new Error('sourceEnd < sourceStart');
-
- // Copy 0 bytes; we're done
- if (end === start) return 0;
- if (target.length == 0 || source.length == 0) return 0;
-
- if (target_start < 0 || target_start >= target.length) {
- throw new Error('targetStart out of bounds');
- }
-
- if (start < 0 || start >= source.length) {
- throw new Error('sourceStart out of bounds');
- }
-
- if (end < 0 || end > source.length) {
- throw new Error('sourceEnd out of bounds');
- }
-
- // Are we oob?
- if (end > this.length) {
- end = this.length;
+ dst[i + offset] = src[i];
+ i++;
}
+ return i;
+}
- if (target.length - target_start < end - start) {
- end = target.length - target_start + start;
+function decodeUtf8Char(str) {
+ try {
+ return decodeURIComponent(str);
+ } catch (err) {
+ return String.fromCharCode(0xFFFD); // UTF 8 invalid char
}
+}
- return this.parent.copy(target.parent,
- target_start + target.offset,
- start + this.offset,
- end + this.offset);
-};
-
-
-// slice(start, end)
-Buffer.prototype.slice = function(start, end) {
- if (end === undefined) end = this.length;
- if (end > this.length) throw new Error('oob');
- if (start > end) throw new Error('oob');
-
- return new Buffer(this.parent, end - start, +start + this.offset);
-};
-
-
-// Legacy methods for backwards compatibility.
-
-Buffer.prototype.utf8Slice = function(start, end) {
- return this.toString('utf8', start, end);
-};
-
-Buffer.prototype.binarySlice = function(start, end) {
- return this.toString('binary', start, end);
-};
-
-Buffer.prototype.asciiSlice = function(start, end) {
- return this.toString('ascii', start, end);
-};
-
-Buffer.prototype.utf8Write = function(string, offset) {
- return this.write(string, offset, 'utf8');
-};
-
-Buffer.prototype.binaryWrite = function(string, offset) {
- return this.write(string, offset, 'binary');
-};
-
-Buffer.prototype.asciiWrite = function(string, offset) {
- return this.write(string, offset, 'ascii');
-};
+// read/write bit-twiddling
Buffer.prototype.readUInt8 = function(offset, noAssert) {
var buffer = this;
@@ -714,7 +497,7 @@ Buffer.prototype.readUInt8 = function(offset, noAssert) {
if (offset >= buffer.length) return;
- return buffer.parent[buffer.offset + offset];
+ return buffer[offset];
};
function readUInt16(buffer, offset, isBigEndian, noAssert) {
@@ -735,14 +518,14 @@ function readUInt16(buffer, offset, isBigEndian, noAssert) {
if (offset >= buffer.length) return 0;
if (isBigEndian) {
- val = buffer.parent[buffer.offset + offset] << 8;
+ val = buffer[offset] << 8;
if (offset + 1 < buffer.length) {
- val |= buffer.parent[buffer.offset + offset + 1];
+ val |= buffer[offset + 1];
}
} else {
- val = buffer.parent[buffer.offset + offset];
+ val = buffer[offset];
if (offset + 1 < buffer.length) {
- val |= buffer.parent[buffer.offset + offset + 1] << 8;
+ val |= buffer[offset + 1] << 8;
}
}
@@ -775,20 +558,20 @@ function readUInt32(buffer, offset, isBigEndian, noAssert) {
if (isBigEndian) {
if (offset + 1 < buffer.length)
- val = buffer.parent[buffer.offset + offset + 1] << 16;
+ val = buffer[offset + 1] << 16;
if (offset + 2 < buffer.length)
- val |= buffer.parent[buffer.offset + offset + 2] << 8;
+ val |= buffer[offset + 2] << 8;
if (offset + 3 < buffer.length)
- val |= buffer.parent[buffer.offset + offset + 3];
- val = val + (buffer.parent[buffer.offset + offset] << 24 >>> 0);
+ val |= buffer[offset + 3];
+ val = val + (buffer[offset] << 24 >>> 0);
} else {
if (offset + 2 < buffer.length)
- val = buffer.parent[buffer.offset + offset + 2] << 16;
+ val = buffer[offset + 2] << 16;
if (offset + 1 < buffer.length)
- val |= buffer.parent[buffer.offset + offset + 1] << 8;
- val |= buffer.parent[buffer.offset + offset];
+ val |= buffer[offset + 1] << 8;
+ val |= buffer[offset];
if (offset + 3 < buffer.length)
- val = val + (buffer.parent[buffer.offset + offset + 3] << 24 >>> 0);
+ val = val + (buffer[offset + 3] << 24 >>> 0);
}
return val;
@@ -862,12 +645,12 @@ Buffer.prototype.readInt8 = function(offset, noAssert) {
if (offset >= buffer.length) return;
- neg = buffer.parent[buffer.offset + offset] & 0x80;
+ neg = buffer[offset] & 0x80;
if (!neg) {
- return (buffer.parent[buffer.offset + offset]);
+ return (buffer[offset]);
}
- return ((0xff - buffer.parent[buffer.offset + offset] + 1) * -1);
+ return ((0xff - buffer[offset] + 1) * -1);
};
function readInt16(buffer, offset, isBigEndian, noAssert) {
@@ -1013,7 +796,7 @@ Buffer.prototype.writeUInt8 = function(value, offset, noAssert) {
}
if (offset < buffer.length) {
- buffer.parent[buffer.offset + offset] = value;
+ buffer[offset] = value;
}
};
@@ -1035,7 +818,7 @@ function writeUInt16(buffer, value, offset, isBigEndian, noAssert) {
}
for (var i = 0; i < Math.min(buffer.length - offset, 2); i++) {
- buffer.parent[buffer.offset + offset + i] =
+ buffer[offset + i] =
(value & (0xff << (8 * (isBigEndian ? 1 - i : i)))) >>>
(isBigEndian ? 1 - i : i) * 8;
}
@@ -1068,7 +851,7 @@ function writeUInt32(buffer, value, offset, isBigEndian, noAssert) {
}
for (var i = 0; i < Math.min(buffer.length - offset, 4); i++) {
- buffer.parent[buffer.offset + offset + i] =
+ buffer[offset + i] =
(value >>> (isBigEndian ? 3 - i : i) * 8) & 0xff;
}
}
@@ -1286,32 +1069,3 @@ Buffer.prototype.writeDoubleLE = function(value, offset, noAssert) {
Buffer.prototype.writeDoubleBE = function(value, offset, noAssert) {
writeDouble(this, value, offset, true, noAssert);
};
-
-SlowBuffer.prototype.readUInt8 = Buffer.prototype.readUInt8;
-SlowBuffer.prototype.readUInt16LE = Buffer.prototype.readUInt16LE;
-SlowBuffer.prototype.readUInt16BE = Buffer.prototype.readUInt16BE;
-SlowBuffer.prototype.readUInt32LE = Buffer.prototype.readUInt32LE;
-SlowBuffer.prototype.readUInt32BE = Buffer.prototype.readUInt32BE;
-SlowBuffer.prototype.readInt8 = Buffer.prototype.readInt8;
-SlowBuffer.prototype.readInt16LE = Buffer.prototype.readInt16LE;
-SlowBuffer.prototype.readInt16BE = Buffer.prototype.readInt16BE;
-SlowBuffer.prototype.readInt32LE = Buffer.prototype.readInt32LE;
-SlowBuffer.prototype.readInt32BE = Buffer.prototype.readInt32BE;
-SlowBuffer.prototype.readFloatLE = Buffer.prototype.readFloatLE;
-SlowBuffer.prototype.readFloatBE = Buffer.prototype.readFloatBE;
-SlowBuffer.prototype.readDoubleLE = Buffer.prototype.readDoubleLE;
-SlowBuffer.prototype.readDoubleBE = Buffer.prototype.readDoubleBE;
-SlowBuffer.prototype.writeUInt8 = Buffer.prototype.writeUInt8;
-SlowBuffer.prototype.writeUInt16LE = Buffer.prototype.writeUInt16LE;
-SlowBuffer.prototype.writeUInt16BE = Buffer.prototype.writeUInt16BE;
-SlowBuffer.prototype.writeUInt32LE = Buffer.prototype.writeUInt32LE;
-SlowBuffer.prototype.writeUInt32BE = Buffer.prototype.writeUInt32BE;
-SlowBuffer.prototype.writeInt8 = Buffer.prototype.writeInt8;
-SlowBuffer.prototype.writeInt16LE = Buffer.prototype.writeInt16LE;
-SlowBuffer.prototype.writeInt16BE = Buffer.prototype.writeInt16BE;
-SlowBuffer.prototype.writeInt32LE = Buffer.prototype.writeInt32LE;
-SlowBuffer.prototype.writeInt32BE = Buffer.prototype.writeInt32BE;
-SlowBuffer.prototype.writeFloatLE = Buffer.prototype.writeFloatLE;
-SlowBuffer.prototype.writeFloatBE = Buffer.prototype.writeFloatBE;
-SlowBuffer.prototype.writeDoubleLE = Buffer.prototype.writeDoubleLE;
-SlowBuffer.prototype.writeDoubleBE = Buffer.prototype.writeDoubleBE;
View
16 package.json
@@ -11,12 +11,24 @@
"base64-js" : "0.0.2"
},
"devDependencies" : {
- "tap" : "0.2.x"
+ "tape" : "~0.3.0",
+ "tap" : "~0.4.0"
},
"repository" : {
"type" : "git",
"url" : "http://github.com/toots/buffer-browserify.git"
},
+ "testling" : {
+ "files" : "test/*.js",
+ "browsers" : [
+ "ie/6..latest",
+ "chrome/20..latest",
+ "firefox/10..latest",
+ "safari/latest",
+ "opera/11.0..latest",
+ "iphone/6", "ipad/6"
+ ]
+ },
"keywords" : [
"buffer",
"browserify",
@@ -29,7 +41,7 @@
"email" : "toots@rastageeks.org"
},
"scripts" : {
- "test" : "node node_modules/tap/bin/tap.js test/*.js"
+ "test" : "tap test/*.js"
},
"license" : "MIT/X11",
"engine" : { "node" : ">=0.6" }
View
111 test/buffer.js
@@ -1,11 +1,11 @@
-var buffer = require('../index.js');
-var test = require('tap').test;
+var B = require('../index.js').Buffer;
+var test = require('tape');
test('utf8 buffer to base64', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("Ձאab", "utf8").toString("base64"),
- new Buffer("Ձאab", "utf8").toString("base64")
+ new B("Ձאab", "utf8").toString("base64"),
+ '1YHXkGFi'
);
t.end();
});
@@ -13,8 +13,8 @@ test('utf8 buffer to base64', function (t) {
test('utf8 buffer to hex', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("Ձאab", "utf8").toString("hex"),
- new Buffer("Ձאab", "utf8").toString("hex")
+ new B("Ձאab", "utf8").toString("hex"),
+ 'd581d7906162'
);
t.end();
});
@@ -22,8 +22,8 @@ test('utf8 buffer to hex', function (t) {
test('utf8 to utf8', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("öäüõÖÄÜÕ", "utf8").toString("utf8"),
- new Buffer("öäüõÖÄÜÕ", "utf8").toString("utf8")
+ new B("öäüõÖÄÜÕ", "utf8").toString("utf8"),
+ 'öäüõÖÄÜÕ'
);
t.end();
});
@@ -31,8 +31,8 @@ test('utf8 to utf8', function (t) {
test('ascii buffer to base64', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("123456!@#$%^", "ascii").toString("base64"),
- new Buffer("123456!@#$%^", "ascii").toString("base64")
+ new B("123456!@#$%^", "ascii").toString("base64"),
+ 'MTIzNDU2IUAjJCVe'
);
t.end();
});
@@ -40,8 +40,8 @@ test('ascii buffer to base64', function (t) {
test('ascii buffer to hex', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("123456!@#$%^", "ascii").toString("hex"),
- new Buffer("123456!@#$%^", "ascii").toString("hex")
+ new B("123456!@#$%^", "ascii").toString("hex"),
+ '31323334353621402324255e'
);
t.end();
});
@@ -49,8 +49,8 @@ test('ascii buffer to hex', function (t) {
test('base64 buffer to utf8', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("1YHXkGFi", "base64").toString("utf8"),
- new Buffer("1YHXkGFi", "base64").toString("utf8")
+ new B("1YHXkGFi", "base64").toString("utf8"),
+ 'Ձאab'
);
t.end();
});
@@ -58,8 +58,8 @@ test('base64 buffer to utf8', function (t) {
test('hex buffer to utf8', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("d581d7906162", "hex").toString("utf8"),
- new Buffer("d581d7906162", "hex").toString("utf8")
+ new B("d581d7906162", "hex").toString("utf8"),
+ 'Ձאab'
);
t.end();
});
@@ -67,8 +67,8 @@ test('hex buffer to utf8', function (t) {
test('base64 buffer to ascii', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("MTIzNDU2IUAjJCVe", "base64").toString("ascii"),
- new Buffer("MTIzNDU2IUAjJCVe", "base64").toString("ascii")
+ new B("MTIzNDU2IUAjJCVe", "base64").toString("ascii"),
+ '123456!@#$%^'
);
t.end();
});
@@ -76,8 +76,8 @@ test('base64 buffer to ascii', function (t) {
test('hex buffer to ascii', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("31323334353621402324255e", "hex").toString("ascii"),
- new Buffer("31323334353621402324255e", "hex").toString("ascii")
+ new B("31323334353621402324255e", "hex").toString("ascii"),
+ '123456!@#$%^'
);
t.end();
});
@@ -85,7 +85,7 @@ test('hex buffer to ascii', function (t) {
test('utf8 to ascii', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("öäüõÖÄÜÕ", "utf8").toString("ascii"),
+ new B("öäüõÖÄÜÕ", "utf8").toString("ascii"),
new Buffer("öäüõÖÄÜÕ", "utf8").toString("ascii")
);
t.end();
@@ -95,8 +95,8 @@ test('utf8 to ascii', function (t) {
test('base64 buffer to binary', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("MTIzNDU2IUAjJCVe", "base64").toString("binary"),
- new Buffer("MTIzNDU2IUAjJCVe", "base64").toString("binary")
+ new B("MTIzNDU2IUAjJCVe", "base64").toString("binary"),
+ '123456!@#$%^'
);
t.end();
});
@@ -104,8 +104,8 @@ test('base64 buffer to binary', function (t) {
test('hex buffer to binary', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("31323334353621402324255e", "hex").toString("binary"),
- new Buffer("31323334353621402324255e", "hex").toString("binary")
+ new B("31323334353621402324255e", "hex").toString("binary"),
+ '123456!@#$%^'
);
t.end();
});
@@ -113,32 +113,35 @@ test('hex buffer to binary', function (t) {
test('utf8 to binary', function (t) {
t.plan(1);
t.equal(
- new buffer.Buffer("öäüõÖÄÜÕ", "utf8").toString("binary"),
- new Buffer("öäüõÖÄÜÕ", "utf8").toString("binary")
+ new B("öäüõÖÄÜÕ", "utf8").toString("binary"),
+ "öäüõÖÄÜÕ"
);
t.end();
});
test("hex of write{Uint,Int}{8,16,32}{LE,BE}", function (t) {
t.plan(2*(2*2*2+2));
+ var hex = [
+ "03", "0300", "0003", "03000000", "00000003",
+ "fd", "fdff", "fffd", "fdffffff", "fffffffd"
+ ];
+ var reads = [ 3, 3, 3, 3, 3, -3, -3, -3, -3, -3 ];
["UInt","Int"].forEach(function(x){
[8,16,32].forEach(function(y){
var endianesses = (y === 8) ? [""] : ["LE","BE"];
endianesses.forEach(function(z){
- var v1 = new buffer.Buffer(y / 8);
- var v2 = new Buffer(y / 8);
+ var v1 = new B(y / 8);
var writefn = "write" + x + y + z;
var val = (x === "Int") ? -3 : 3;
v1[writefn](val, 0);
- v2[writefn](val, 0);
t.equal(
v1.toString("hex"),
- v2.toString("hex")
+ hex.shift()
);
var readfn = "read" + x + y + z;
t.equal(
v1[readfn](0),
- v2[readfn](0)
+ reads.shift()
);
});
});
@@ -148,21 +151,27 @@ test("hex of write{Uint,Int}{8,16,32}{LE,BE}", function (t) {
test("hex of write{Uint,Int}{8,16,32}{LE,BE} with overflow", function (t) {
t.plan(3*(2*2*2+2));
+ var hex = [
+ "", "03", "00", "030000", "000000",
+ "", "fd", "ff", "fdffff", "ffffff"
+ ];
+ var reads = [
+ undefined, 3, 0, 3, 0,
+ undefined, 253, -256, 16777213, -256
+ ];
["UInt","Int"].forEach(function(x){
[8,16,32].forEach(function(y){
var endianesses = (y === 8) ? [""] : ["LE","BE"];
endianesses.forEach(function(z){
- var v1 = new buffer.Buffer(y / 8 - 1);
- var v2 = new Buffer(y / 8 - 1);
- var next = new buffer.Buffer(4);
+ var v1 = new B(y / 8 - 1);
+ var next = new B(4);
next.writeUInt32BE(0, 0);
var writefn = "write" + x + y + z;
var val = (x === "Int") ? -3 : 3;
v1[writefn](val, 0, true);
- v2[writefn](val, 0, true);
t.equal(
v1.toString("hex"),
- v2.toString("hex")
+ hex.shift()
);
// check that nothing leaked to next buffer.
t.equal(next.readUInt32BE(0), 0);
@@ -171,7 +180,7 @@ test("hex of write{Uint,Int}{8,16,32}{LE,BE} with overflow", function (t) {
var readfn = "read" + x + y + z;
t.equal(
v1[readfn](0, true),
- v2[readfn](0, true)
+ reads.shift()
);
});
});
@@ -182,14 +191,14 @@ test("hex of write{Uint,Int}{8,16,32}{LE,BE} with overflow", function (t) {
test("concat() a varying number of buffers", function (t) {
t.plan(5);
var zero = [];
- var one = [ new buffer.Buffer('asdf') ];
+ var one = [ new B('asdf') ];
var long = [];
- for (var i = 0; i < 10; i++) long.push(new buffer.Buffer('asdf'));
+ for (var i = 0; i < 10; i++) long.push(new B('asdf'));
- var flatZero = buffer.Buffer.concat(zero);
- var flatOne = buffer.Buffer.concat(one);
- var flatLong = buffer.Buffer.concat(long);
- var flatLongLen = buffer.Buffer.concat(long, 40);
+ var flatZero = B.concat(zero);
+ var flatOne = B.concat(one);
+ var flatLong = B.concat(long);
+ var flatLongLen = B.concat(long, 40);
t.equal(flatZero.length, 0);
t.equal(flatOne.toString(), 'asdf');
@@ -201,18 +210,16 @@ test("concat() a varying number of buffers", function (t) {
test("buffer from buffer", function (t) {
t.plan(1);
- var b1 = new buffer.Buffer('asdf');
- var b2 = new buffer.Buffer(b1);
+ var b1 = new B('asdf');
+ var b2 = new B(b1);
t.equal(b1.toString('hex'), b2.toString('hex'));
t.end();
});
test("fill", function(t) {
t.plan(1);
- var b1 = new Buffer(10);
- var b2 = new buffer.Buffer(10);
- b1.fill(2);
- b2.fill(2);
- t.equal(b1.toString('hex'), b2.toString('hex'));
+ var b = new B(10);
+ b.fill(2);
+ t.equal(b.toString('hex'), '02020202020202020202');
t.end();
-})
+});
View
29 test/indexes.js
@@ -0,0 +1,29 @@
+var B = require('../').Buffer;
+var test = require('tape');
+
+test('indexes from a string', function(t) {
+ t.plan(3);
+ var buf = new B('abc');
+ t.equal(buf[0], 97);
+ t.equal(buf[1], 98);
+ t.equal(buf[2], 99);
+});
+
+test('indexes from an array', function(t) {
+ t.plan(3);
+ var buf = new B([ 97, 98, 99 ]);
+ t.equal(buf[0], 97);
+ t.equal(buf[1], 98);
+ t.equal(buf[2], 99);
+});
+
+test('set then modify indexes from an array', function(t) {
+ t.plan(4);
+ var buf = new B([ 97, 98, 99 ]);
+ t.equal(buf[2], 99);
+ t.equal(buf.toString(), 'abc');
+
+ buf[2] += 10;
+ t.equal(buf[2], 109);
+ t.equal(buf.toString(), 'abm');
+});
Something went wrong with that request. Please try again.