New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using sun.misc for Base64Binary does not handle CRLF in images #173

Closed
nmccready opened this Issue Nov 7, 2012 · 13 comments

Comments

Projects
None yet
3 participants
@nmccready

Here are the unit tests.
Base64Spec here - https://gist.github.com/4032388
The resource image is here - https://www.sugarsync.com/pf/D279748_79_8869386261.

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Nov 7, 2012

Owner

I haven't tried your code yet, but here's from XML Schema Part 2:

Base64Binary  ::=  ((B64S B64S B64S B64S)*
                     ((B64S B64S B64S B64) |
                      (B64S B64S B16S '=') |
                      (B64S B04S '=' #x20? '=')))?

B64S         ::= B64 #x20?

B16S         ::= B16 #x20?

B04S         ::= B04 #x20? 

B04         ::=  [AQgw]
B16         ::=  [AEIMQUYcgkosw048]
B64         ::=  [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.

Owner

eed3si9n commented Nov 7, 2012

I haven't tried your code yet, but here's from XML Schema Part 2:

Base64Binary  ::=  ((B64S B64S B64S B64S)*
                     ((B64S B64S B64S B64) |
                      (B64S B64S B16S '=') |
                      (B64S B04S '=' #x20? '=')))?

B64S         ::= B64 #x20?

B16S         ::= B16 #x20?

B04S         ::= B04 #x20? 

B04         ::=  [AQgw]
B16         ::=  [AEIMQUYcgkosw048]
B64         ::=  [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Nov 7, 2012

Hmm, well for the most part your stuff works as long as you stip out
whitespace and CRLF ahead of time. I first saw these errors when trying to
save an image from a soap msg encoded in base64. The save ended up being
junky. It appears to me that apache handles the variations easier. If you
don't use Apache no skin off my nose, I'll just pass it as a string in the
xsd instead.

Either way one should be chosen that has the least amount of work arounds
to make it work.

Nick

On Wed, Nov 7, 2012 at 11:10 AM, eugene yokota notifications@github.comwrote:

I haven't tried your code yet, but here's from XML Schema Part 2http://www.w3.org/TR/xmlschema-2/#base64Binary
:

Base64Binary ::= ((B64S B64S B64S B64S)*
((B64S B64S B64S B64) |
(B64S B64S B16S '=') |
(B64S B04S '=' #x20? '=')))?

B64S ::= B64 #x20?

B16S ::= B16 #x20?

B04S ::= B04 #x20?

B04 ::= [AQgw]
B16 ::= [AEIMQUYcgkosw048]
B64 ::= [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10153821.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Hmm, well for the most part your stuff works as long as you stip out
whitespace and CRLF ahead of time. I first saw these errors when trying to
save an image from a soap msg encoded in base64. The save ended up being
junky. It appears to me that apache handles the variations easier. If you
don't use Apache no skin off my nose, I'll just pass it as a string in the
xsd instead.

Either way one should be chosen that has the least amount of work arounds
to make it work.

Nick

On Wed, Nov 7, 2012 at 11:10 AM, eugene yokota notifications@github.comwrote:

I haven't tried your code yet, but here's from XML Schema Part 2http://www.w3.org/TR/xmlschema-2/#base64Binary
:

Base64Binary ::= ((B64S B64S B64S B64S)*
((B64S B64S B64S B64) |
(B64S B64S B16S '=') |
(B64S B04S '=' #x20? '=')))?

B64S ::= B64 #x20?

B16S ::= B16 #x20?

B04S ::= B04 #x20?

B04 ::= [AQgw]
B16 ::= [AEIMQUYcgkosw048]
B64 ::= [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10153821.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Nov 7, 2012

FYI I am using 0.8.8, I'll have to upgrade later as I am moving on to
pushing my data to repositories now.

On Wed, Nov 7, 2012 at 11:19 AM, Nick McCready nmccready@gmail.com wrote:

Hmm, well for the most part your stuff works as long as you stip out
whitespace and CRLF ahead of time. I first saw these errors when trying to
save an image from a soap msg encoded in base64. The save ended up being
junky. It appears to me that apache handles the variations easier. If you
don't use Apache no skin off my nose, I'll just pass it as a string in the
xsd instead.

Either way one should be chosen that has the least amount of work arounds
to make it work.

Nick

On Wed, Nov 7, 2012 at 11:10 AM, eugene yokota notifications@github.comwrote:

I haven't tried your code yet, but here's from XML Schema Part 2http://www.w3.org/TR/xmlschema-2/#base64Binary
:

Base64Binary ::= ((B64S B64S B64S B64S)*
((B64S B64S B64S B64) |
(B64S B64S B16S '=') |
(B64S B04S '=' #x20? '=')))?

B64S ::= B64 #x20?

B16S ::= B16 #x20?

B04S ::= B04 #x20?

B04 ::= [AQgw]
B16 ::= [AEIMQUYcgkosw048]
B64 ::= [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10153821.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

FYI I am using 0.8.8, I'll have to upgrade later as I am moving on to
pushing my data to repositories now.

On Wed, Nov 7, 2012 at 11:19 AM, Nick McCready nmccready@gmail.com wrote:

Hmm, well for the most part your stuff works as long as you stip out
whitespace and CRLF ahead of time. I first saw these errors when trying to
save an image from a soap msg encoded in base64. The save ended up being
junky. It appears to me that apache handles the variations easier. If you
don't use Apache no skin off my nose, I'll just pass it as a string in the
xsd instead.

Either way one should be chosen that has the least amount of work arounds
to make it work.

Nick

On Wed, Nov 7, 2012 at 11:10 AM, eugene yokota notifications@github.comwrote:

I haven't tried your code yet, but here's from XML Schema Part 2http://www.w3.org/TR/xmlschema-2/#base64Binary
:

Base64Binary ::= ((B64S B64S B64S B64S)*
((B64S B64S B64S B64) |
(B64S B64S B16S '=') |
(B64S B04S '=' #x20? '=')))?

B64S ::= B64 #x20?

B16S ::= B16 #x20?

B04S ::= B04 #x20?

B04 ::= [AQgw]
B16 ::= [AEIMQUYcgkosw048]
B64 ::= [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10153821.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Nov 7, 2012

Owner

I haven't tested the #x20 case, but if scalaxb doesn't handle that, it's a bug.
I am reluctant to start supporting non-legal encodings, but I am undecided. What's 0.8.8? The current version of scalaxb is 1.0.0-RC1 before that was 0.7.3.

Owner

eed3si9n commented Nov 7, 2012

I haven't tested the #x20 case, but if scalaxb doesn't handle that, it's a bug.
I am reluctant to start supporting non-legal encodings, but I am undecided. What's 0.8.8? The current version of scalaxb is 1.0.0-RC1 before that was 0.7.3.

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Nov 7, 2012

Also here attached is the NTLM code I been meaning to blog. I do need to
test it more with authentication, but I am pretty sure it works.

On Wed, Nov 7, 2012 at 11:20 AM, Nick McCready nmccready@gmail.com wrote:

FYI I am using 0.8.8, I'll have to upgrade later as I am moving on to
pushing my data to repositories now.

On Wed, Nov 7, 2012 at 11:19 AM, Nick McCready nmccready@gmail.comwrote:

Hmm, well for the most part your stuff works as long as you stip out
whitespace and CRLF ahead of time. I first saw these errors when trying to
save an image from a soap msg encoded in base64. The save ended up being
junky. It appears to me that apache handles the variations easier. If you
don't use Apache no skin off my nose, I'll just pass it as a string in the
xsd instead.

Either way one should be chosen that has the least amount of work arounds
to make it work.

Nick

On Wed, Nov 7, 2012 at 11:10 AM, eugene yokota notifications@github.comwrote:

I haven't tried your code yet, but here's from XML Schema Part 2http://www.w3.org/TR/xmlschema-2/#base64Binary
:

Base64Binary ::= ((B64S B64S B64S B64S)*
((B64S B64S B64S B64) |
(B64S B64S B16S '=') |
(B64S B04S '=' #x20? '=')))?

B64S ::= B64 #x20?

B16S ::= B16 #x20?

B04S ::= B04 #x20?

B04 ::= [AQgw]
B16 ::= [AEIMQUYcgkosw048]
B64 ::= [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10153821.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Also here attached is the NTLM code I been meaning to blog. I do need to
test it more with authentication, but I am pretty sure it works.

On Wed, Nov 7, 2012 at 11:20 AM, Nick McCready nmccready@gmail.com wrote:

FYI I am using 0.8.8, I'll have to upgrade later as I am moving on to
pushing my data to repositories now.

On Wed, Nov 7, 2012 at 11:19 AM, Nick McCready nmccready@gmail.comwrote:

Hmm, well for the most part your stuff works as long as you stip out
whitespace and CRLF ahead of time. I first saw these errors when trying to
save an image from a soap msg encoded in base64. The save ended up being
junky. It appears to me that apache handles the variations easier. If you
don't use Apache no skin off my nose, I'll just pass it as a string in the
xsd instead.

Either way one should be chosen that has the least amount of work arounds
to make it work.

Nick

On Wed, Nov 7, 2012 at 11:10 AM, eugene yokota notifications@github.comwrote:

I haven't tried your code yet, but here's from XML Schema Part 2http://www.w3.org/TR/xmlschema-2/#base64Binary
:

Base64Binary ::= ((B64S B64S B64S B64S)*
((B64S B64S B64S B64) |
(B64S B64S B16S '=') |
(B64S B04S '=' #x20? '=')))?

B64S ::= B64 #x20?

B16S ::= B16 #x20?

B04S ::= B04 #x20?

B04 ::= [AQgw]
B16 ::= [AEIMQUYcgkosw048]
B64 ::= [A-Za-z0-9+/]

The only legal whitespace in base64Binary is #x20.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10153821.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Nov 7, 2012

Owner

So you're saying Microsoft is the offending party?

Owner

eed3si9n commented Nov 7, 2012

So you're saying Microsoft is the offending party?

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Nov 7, 2012

Sorry I meant 0.7.3 of the plugin. Oops.

On Wed, Nov 7, 2012 at 11:31 AM, eugene yokota notifications@github.comwrote:

I haven't tested the #x20 case, but if scalaxb doesn't handle that, it's
a bug.
I am reluctant to start supporting non-legal encodings, but I am
undecided. What's 0.8.8? The current version of scalaxb is 1.0.0-RC1 before
that was 0.7.3.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10154717.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Sorry I meant 0.7.3 of the plugin. Oops.

On Wed, Nov 7, 2012 at 11:31 AM, eugene yokota notifications@github.comwrote:

I haven't tested the #x20 case, but if scalaxb doesn't handle that, it's
a bug.
I am reluctant to start supporting non-legal encodings, but I am
undecided. What's 0.8.8? The current version of scalaxb is 1.0.0-RC1 before
that was 0.7.3.


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10154717.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Nov 7, 2012

Yes and no, it could be the offending party is the company we are working
with. However I have the same problems not using soap in those unit tests
with the android image. So I do not think it is primarily Microsoft. But
yes I am talking to a MSFT box. Actually the mage coming over is not coming
over via soap.. it is a tcp connection but the client sends xml. The XML is
deserialized by me using scalaxb's xsd classes. The SOAP web service part
is only used to tell their system where to connect as a "router" like
mechanism.

On Wed, Nov 7, 2012 at 11:33 AM, eugene yokota notifications@github.comwrote:

So you're saying Microsoft is the offending party?


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10154790.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

Yes and no, it could be the offending party is the company we are working
with. However I have the same problems not using soap in those unit tests
with the android image. So I do not think it is primarily Microsoft. But
yes I am talking to a MSFT box. Actually the mage coming over is not coming
over via soap.. it is a tcp connection but the client sends xml. The XML is
deserialized by me using scalaxb's xsd classes. The SOAP web service part
is only used to tell their system where to connect as a "router" like
mechanism.

On Wed, Nov 7, 2012 at 11:33 AM, eugene yokota notifications@github.comwrote:

So you're saying Microsoft is the offending party?


Reply to this email directly or view it on GitHubhttps://github.com/eed3si9n/scalaxb/issues/173#issuecomment-10154790.

Nicholas McCready

Personal Email: nmccready@gmail.com
Web site: http://codemonkeyseedo.blogspot.com/
Twitter: nmccready https://twitter.com/#!/nmccready

@martiell

This comment has been minimized.

Show comment
Hide comment
@martiell

martiell Nov 7, 2012

Collaborator

I wonder if it's worth using javax.xml.bind.DatatypeConverter#parseBase64Binary instead?

The xml.bind implementation ignores new lines:

scala> new BASE64Decoder().decodeBuffer("AA\nAA")
res0: Array[Byte] = Array(0, 15, -64)

scala> parseBase64Binary("AA\nAA")
res1: Array[Byte] = Array(0, 0, 0)

It also has the advantage that the scalaxb runtime is slightly easier to use in an OSGi environment if there's no dependency on sun.misc.

Collaborator

martiell commented Nov 7, 2012

I wonder if it's worth using javax.xml.bind.DatatypeConverter#parseBase64Binary instead?

The xml.bind implementation ignores new lines:

scala> new BASE64Decoder().decodeBuffer("AA\nAA")
res0: Array[Byte] = Array(0, 15, -64)

scala> parseBase64Binary("AA\nAA")
res1: Array[Byte] = Array(0, 0, 0)

It also has the advantage that the scalaxb runtime is slightly easier to use in an OSGi environment if there's no dependency on sun.misc.

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Nov 7, 2012

Owner

@martiell nice. I like that much better than introducing Apache common deps.

Owner

eed3si9n commented Nov 7, 2012

@martiell nice. I like that much better than introducing Apache common deps.

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Nov 7, 2012

cool, if it passes all the test I am game. I just used apache as it was the first other implementation I came across.

cool, if it passes all the test I am game. I just used apache as it was the first other implementation I came across.

@eed3si9n eed3si9n closed this in a2e1776 Nov 9, 2012

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Nov 11, 2012

Owner

scalaxb 1.0.0-RC2 is out.

Owner

eed3si9n commented Nov 11, 2012

scalaxb 1.0.0-RC2 is out.

@nmccready

This comment has been minimized.

Show comment
Hide comment
@nmccready

nmccready Mar 25, 2013

thanks for the props

thanks for the props

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment