Handle partially parsed objects (w/ querystrings as keys) #27

Open
wants to merge 2 commits into
from

Conversation

Projects
None yet
2 participants
@kainosnoema

tl;dr

  • { 'foo[bar]': { qux: 0 }, 'foo': { baz: 1 } } => { foo: { bar: { qux: 0 }, baz: 1 } }
  • previously this would work in some cases, this fix handles most cases properly
  • allows for handling of pre-parsed multipart forms from felixge/node-formidable
  • only added a few lines plus tests, all tests pass

Why?

For multipart forms, particularly from node-formidable. This particular use case is important:

We can use mimetypes to partially parse data from node-formidable parts, before using qs to reassemble

// Raw POST body:
//
// --xxxxxxxx
// Content-Disposition: form-data; name="foo"
// Content-Type: application/json
// 
// {"bar": "baz"}
// --xxxxxxxx
// Content-Disposition: form-data; name="foo[qux]"
// Content-Type: image/jpeg
// 
// (... image data ...)
// --xxxxxxxx--

// Simplified example:

> console.log(qs.parse({
>   foo: JSON.parse("{\"bar\": \"baz\"}")
> , 'foo[qux]': [object Object] // JPEG data in Formidable.File object
> }));
>
{
  foo: {
    bar: "baz",
    qux: [object Object] // JPEG data in Formidable.File object
  }
}

Does we really need this?

Yes. The RestKit iOS framework (among others) can send multipart forms containing both application/json and attachment (ie. image/jpeg) data. Properly merging the parts back into a single params object is crucial. The ExpressJS body-parser got us most of the way there—although we did have to override handlePart() so we could parse the JSON based on the mimetype.

The final piece of the puzzle is getting qs to reassemble the parts from the querystring paths as expected. As I mentioned in the commit message, qs was doing this properly in some cases, but not all (depending on part order from node-formidable, which is apparently non-deterministic). This fix should handle most cases as expected.

Handle partially parsed objects (w/ querystrings as keys)
- previously this would work in some cases, this fix handles most cases properly
- { 'foo[bar]': { qux: 0 }, 'foo': { baz: 1 } } => { foo: { bar: { qux: 0 }, baz: 1 } }
- allows for handling of pre-parsed multipart forms from felixge/node-formidable
@@ -232,10 +224,13 @@ function stringifyObject(obj, prefix) {
function set(obj, key, val) {
var v = obj[key];
- if (undefined === v) {
+ if ('undefined' == typeof v) {

This comment has been minimized.

Show comment Hide comment
@tj

tj Dec 18, 2011

Owner

why do we need this? where's v coming from that we need typeof

@tj

tj Dec 18, 2011

Owner

why do we need this? where's v coming from that we need typeof

This comment has been minimized.

Show comment Hide comment
@tj

tj Dec 18, 2011

Owner

oh it's right above haha, we dont need typeof then

@tj

tj Dec 18, 2011

Owner

oh it's right above haha, we dont need typeof then

This comment has been minimized.

Show comment Hide comment
@kainosnoema

kainosnoema Dec 18, 2011

Just consistency really, and because undefined can be redefined in Javascript, meaning the safest way to check for undefined is with typeof. It's a habit of mine...

EDIT: I knew this, but just tried it... blows my mind:

> typeof undefined
'undefined'
> {}['foo'] === undefined
true
> undefined = 'foo';
'foo'
> typeof undefined
'string'
> {}['foo'] === undefined
false

EDIT 2: Apparently this behavior was eliminated in newer versions of V8? The above was in Node v0.4.8, v0.6.2 doesn't allow it... thank goodness.

@kainosnoema

kainosnoema Dec 18, 2011

Just consistency really, and because undefined can be redefined in Javascript, meaning the safest way to check for undefined is with typeof. It's a habit of mine...

EDIT: I knew this, but just tried it... blows my mind:

> typeof undefined
'undefined'
> {}['foo'] === undefined
true
> undefined = 'foo';
'foo'
> typeof undefined
'string'
> {}['foo'] === undefined
false

EDIT 2: Apparently this behavior was eliminated in newer versions of V8? The above was in Node v0.4.8, v0.6.2 doesn't allow it... thank goodness.

This comment has been minimized.

Show comment Hide comment
@tj

tj Dec 18, 2011

Owner

ah gotcha, personally I avoid defensive programming whenever possible to delegate the issue. plus there are a ton of retarded things you can do in js, people just shouldn't do them haha

@tj

tj Dec 18, 2011

Owner

ah gotcha, personally I avoid defensive programming whenever possible to delegate the issue. plus there are a ton of retarded things you can do in js, people just shouldn't do them haha

lib/querystring.js
obj[key] = val;
} else if (Array.isArray(v)) {
v.push(val);
+ } else if (('object' == typeof v) && ('[object Object]' == toString.call(val))) {
+ // partially parsed object
+ parseObject(val, v);

This comment has been minimized.

Show comment Hide comment
@kainosnoema

kainosnoema Dec 18, 2011

btw, putting this recursive call inside set() seems a very odd to me given that its not actually setting anything. The only reason is to keep things a little cleaner since you'd otherwise have to check and handle this case in both the parse() and merge() functions. That said, if you have a better idea I'll be glad to make changes.

@kainosnoema

kainosnoema Dec 18, 2011

btw, putting this recursive call inside set() seems a very odd to me given that its not actually setting anything. The only reason is to keep things a little cleaner since you'd otherwise have to check and handle this case in both the parse() and merge() functions. That said, if you have a better idea I'll be glad to make changes.

This comment has been minimized.

Show comment Hide comment
@tj

tj Dec 18, 2011

Owner

meh yeah it's, the whole lib is pretty obscure, it's a strange thing to parse, as long as we have strong test cases we should be fine

@tj

tj Dec 18, 2011

Owner

meh yeah it's, the whole lib is pretty obscure, it's a strange thing to parse, as long as we have strong test cases we should be fine

@kainosnoema

This comment has been minimized.

Show comment Hide comment
@kainosnoema

kainosnoema Dec 19, 2011

Ok, so I've found a weird edge case that I'd like to get your feedback on before you do anything with this:

Current behavior:

> console.log(qs.parse({ 'foo[items]': [], foo: { items: ['bar'] } }))
{ "foo": { "items": [ [ "bar" ] ] } }

> console.log(qs.parse({ foo: { items: ['bar'] }, 'foo[items]': [] }))
{ "foo": { "items": [ "bar", [] ] } }

Oddly enough, we actually ran into this situation with RestKit (starting to hate it). We were expecting a consistent output, regardless of order, but unfortunately there isn't one when you use the "rules" of querystrings: that repeated keys are pushed together into arrays. The output above actually makes sense given the rules—its just not what we expected or wanted.

One possible solution would be to use concat() instead of push() under the set() function, since with non-Array objects, concat() behaves similarly to push() (so we keep existing behavior for normal querystrings—all tests still pass). At least the result is consistent regardless of order, but again, this isn't standard querystring behavior:

Using concat() instead of push():

> console.log(qs.parse({ 'foo[items]': [], foo: { items: ['bar'] } }))
{ "foo": { "items": [ "bar" ] } }

> console.log(qs.parse({ foo: { items: ['bar'] }, 'foo[items]': [] }))
{ "foo": { "items": [ "bar" ] } }

To be honest, I don't really like either option—this whole thing seems really weird.

Thoughts?

Ok, so I've found a weird edge case that I'd like to get your feedback on before you do anything with this:

Current behavior:

> console.log(qs.parse({ 'foo[items]': [], foo: { items: ['bar'] } }))
{ "foo": { "items": [ [ "bar" ] ] } }

> console.log(qs.parse({ foo: { items: ['bar'] }, 'foo[items]': [] }))
{ "foo": { "items": [ "bar", [] ] } }

Oddly enough, we actually ran into this situation with RestKit (starting to hate it). We were expecting a consistent output, regardless of order, but unfortunately there isn't one when you use the "rules" of querystrings: that repeated keys are pushed together into arrays. The output above actually makes sense given the rules—its just not what we expected or wanted.

One possible solution would be to use concat() instead of push() under the set() function, since with non-Array objects, concat() behaves similarly to push() (so we keep existing behavior for normal querystrings—all tests still pass). At least the result is consistent regardless of order, but again, this isn't standard querystring behavior:

Using concat() instead of push():

> console.log(qs.parse({ 'foo[items]': [], foo: { items: ['bar'] } }))
{ "foo": { "items": [ "bar" ] } }

> console.log(qs.parse({ foo: { items: ['bar'] }, 'foo[items]': [] }))
{ "foo": { "items": [ "bar" ] } }

To be honest, I don't really like either option—this whole thing seems really weird.

Thoughts?

@tj

This comment has been minimized.

Show comment Hide comment
@tj

tj Dec 19, 2011

Owner

my vote is on the second I think

Owner

tj commented Dec 19, 2011

my vote is on the second I think

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment