UTF-8 Javascipt is not served properly in development mode

[crystalcyril]

I have been using Struts2 and recently switched to Grails 2.3.1.

I have tried using asset pipeline version 1.5.3. However, when I am trying to serving a Javascript library which contains Unicode characters, I found that it does not properly serve the Javacsript file.

Environment

• Operating system: Windows 7 64-bit
• Grails: 2.3.1
• asset-pipeline plugin: 1.5.3
• Javascript library: momentjs. (URL: http://momentjs.com/downloads/moment-with-langs.js)

How to reproduce

1. create a new grails project using command "grails create-app"
2. copy the momentjs javascript file to the grails-app/asset/javascripts/moment-with-langs.js
3. in the GSP file, use the <asset:javascript src="moment-with-langs.js" />
4. in grails console, enter "run-app" and visit the URL of the javascript. e.g. http://localhost:8080/foobar/asset/moment-with-langs.js?compile=false
5. search for the keyword "japan"
6. You can see that the unicode character is currupted.

Things to note

• The same Javascript file is used in my old Apache Struts2 framework without any modification.

What I have tried

• I tried specifying character encoding in the tag e.g. <asset:javascript charset="utf-8" >. But it does not work
• I have tried modifying the momentjs javascript source to DOS/UNIX format as well as playing around with UTF-8 and Unicode format.

Thoughts and Possible Solution

Obviously it is something related to character encoding, however it can happen in several areas

• Reading the source Javascript file.
• Serving to user agent via controller.
• Tomcat connector.

After some tracing I found that the Grails controller AssetsController maybe the cause of problem.

Git commit: 3266b01
Source file: https://github.com/bertramdev/asset-pipeline/blob/3266b0168219723753b617c1ea391c4469ec68e8/grails-app/controllers/asset/pipeline/AssetsController.groovy

if(assetFile) {
  response.setContentType(format)
  response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
  response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
  response.setDateHeader("Expires", 0); // Proxies.
  if(format == 'text/html') {
    render contentType: 'text/html', text: new String(assetFile)
  } else {
    response.outputStream << assetFile
    response.flushBuffer()
}

The code "response.outputStream << assetFile" seems not specifying any encoding.

So I made some minor changes to:

1. specify HTTP response character encoding if "encoding" parameter is specified.
2. use HttpResponse.getWriter() to write instead of outputStream.

Here is the code fix: (marked with // FIX between)

if(assetFile) {
  response.setContentType(format)
  response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
  response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
  response.setDateHeader("Expires", 0); // Proxies.
  // FIX begins
  if (params.encoding != null) {
    response.setCharacterEncoding(params.encoding);
  }
  // FIX ends
  if(format == 'text/html') {
    render contentType: 'text/html', text: new String(assetFile)
  } else {
    // FIX begins
    if (params.encoding != null) {
      def w = response.writer
      w.write(assetFile)
      w.flush()
    } else {
      response.outputStream << assetFile
      response.flushBuffer()
    }
    // FIX ends
}

[davydotcom]

Ah ok you will need charset I'll get this in and a test case in place to make sure this doesn't come back :) it seems jdk7 on OS X switched default encoding to utf-8 so it wasn't appearing.

Sent from my iPhone

On Feb 19, 2014, at 6:42 AM, Cyril Poon notifications@github.com wrote:

I have been using Struts2 and recently switched to Grails 2.3.1.

I have tried using asset pipeline version 1.5.3. However, when I am trying to serving a Javascript library which contains Unicode characters, I found that it does not properly serve the Javacsript file.

Environment

Operating system: Windows 7 64-bit
Grails: 2.3.1
asset-pipeline plugin: 1.5.3
Javascript library: momentjs. (URL: http://momentjs.com/downloads/moment-with-langs.js)
How to reproduce

create a new grails project using command "grails create-app"
copy the momentjs javascript file to the grails-app/asset/javascripts/moment-with-langs.js
in the GSP file, use the
in grails console, enter "run-app" and visit the URL of the javascript. e.g. http://localhost:8080/foobar/asset/moment-with-langs.js?compile=false
search for the keyword "japan"
You can see that the unicode character is currupted.
Things to note

The same Javascript file is used in my old Apache Struts2 framework without any modification.
What I have tried

I tried specifying character encoding in the tag e.g. . But it does not work
I have tried modifying the momentjs javascript source to DOS/UNIX format as well as playing around with UTF-8 and Unicode format.
Thoughts and Possible Solution

Obviously it is something related to character encoding, however it can happen in several areas

Reading the source Javascript file.
Serving to user agent via controller.
Tomcat connector.
After some tracing I found that the Grails controller AssetsController maybe the cause of problem.

Git commit: 3266b01
Source file: https://github.com/bertramdev/asset-pipeline/blob/3266b0168219723753b617c1ea391c4469ec68e8/grails-app/controllers/asset/pipeline/AssetsController.groovy

if(assetFile) {
response.setContentType(format)
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setDateHeader("Expires", 0); // Proxies.
if(format == 'text/html') {
render contentType: 'text/html', text: new String(assetFile)
} else {
response.outputStream << assetFile
response.flushBuffer()
}
The code "response.outputStream << assetFile" seems not specifying any encoding.

So I made some minor changes to:

specify HTTP response character encoding if "encoding" parameter is specified.
use HttpResponse.getWriter() to write instead of outputStream.
Here is the code fix: (marked with // FIX between)

if(assetFile) {
response.setContentType(format)
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setDateHeader("Expires", 0); // Proxies.
// FIX begins
if (params.encoding != null) {
response.setCharacterEncoding(params.encoding);
}
// FIX ends
if(format == 'text/html') {
render contentType: 'text/html', text: new String(assetFile)
} else {
// FIX begins
if (params.encoding != null) {
def w = response.writer
w.write(assetFile)
w.flush()
} else {
response.outputStream << assetFile
response.flushBuffer()
}
// FIX ends
}
—
Reply to this email directly or view it on GitHub.

[davydotcom]

Got it couple things, It was simply a matter of setting the response character encoding. But note on encoding differences between development and war/production.

In development, encode detection is handled by both the //=encoding directive and the charset=utf8 attribute. Make sure you add //=encoding utf8 on the root javascript file you plan on using (your js manifest). This way when the assets are precompiled for production the system has a way to infer the desired encoding for compilation.

You should be able to give this a shot in asset-pipeline version 1.5.4

Thanks,
David

On Feb 19, 2014, at 7:11 AM, David Estes davydotcom@gmail.com wrote:

Ah ok you will need charset I'll get this in and a test case in place to make sure this doesn't come back :) it seems jdk7 on OS X switched default encoding to utf-8 so it wasn't appearing.

Sent from my iPhone

On Feb 19, 2014, at 6:42 AM, Cyril Poon notifications@github.com wrote:

I have been using Struts2 and recently switched to Grails 2.3.1.

I have tried using asset pipeline version 1.5.3. However, when I am trying to serving a Javascript library which contains Unicode characters, I found that it does not properly serve the Javacsript file.

Environment

Operating system: Windows 7 64-bit
Grails: 2.3.1
asset-pipeline plugin: 1.5.3
Javascript library: momentjs. (URL: http://momentjs.com/downloads/moment-with-langs.js)
How to reproduce

create a new grails project using command "grails create-app"
copy the momentjs javascript file to the grails-app/asset/javascripts/moment-with-langs.js
in the GSP file, use the
in grails console, enter "run-app" and visit the URL of the javascript. e.g. http://localhost:8080/foobar/asset/moment-with-langs.js?compile=false
search for the keyword "japan"
You can see that the unicode character is currupted.
Things to note

The same Javascript file is used in my old Apache Struts2 framework without any modification.
What I have tried

I tried specifying character encoding in the tag e.g. . But it does not work
I have tried modifying the momentjs javascript source to DOS/UNIX format as well as playing around with UTF-8 and Unicode format.
Thoughts and Possible Solution

Obviously it is something related to character encoding, however it can happen in several areas

Reading the source Javascript file.
Serving to user agent via controller.
Tomcat connector.
After some tracing I found that the Grails controller AssetsController maybe the cause of problem.

Git commit: 3266b01
Source file: https://github.com/bertramdev/asset-pipeline/blob/3266b0168219723753b617c1ea391c4469ec68e8/grails-app/controllers/asset/pipeline/AssetsController.groovy

if(assetFile) {
response.setContentType(format)
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setDateHeader("Expires", 0); // Proxies.
if(format == 'text/html') {
render contentType: 'text/html', text: new String(assetFile)
} else {
response.outputStream << assetFile
response.flushBuffer()
}
The code "response.outputStream << assetFile" seems not specifying any encoding.

So I made some minor changes to:

specify HTTP response character encoding if "encoding" parameter is specified.
use HttpResponse.getWriter() to write instead of outputStream.
Here is the code fix: (marked with // FIX between)

if(assetFile) {
response.setContentType(format)
response.setHeader("Cache-Control", "no-cache, no-store, must-revalidate"); // HTTP 1.1.
response.setHeader("Pragma", "no-cache"); // HTTP 1.0.
response.setDateHeader("Expires", 0); // Proxies.
// FIX begins
if (params.encoding != null) {
response.setCharacterEncoding(params.encoding);
}
// FIX ends
if(format == 'text/html') {
render contentType: 'text/html', text: new String(assetFile)
} else {
// FIX begins
if (params.encoding != null) {
def w = response.writer
w.write(assetFile)
w.flush()
} else {
response.outputStream << assetFile
response.flushBuffer()
}
// FIX ends
}
—
Reply to this email directly or view it on GitHub.

[davydotcom]

Resolved