Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

International keywords #487

Closed
hangvar opened this Issue · 41 comments

5 participants

@hangvar

Hi,

I have problems switching the examples to Swedish international keywords. I've tried helloworld as well as java-calculator and both are having problems when I change to the following import statements:

import cucumber.api.java.sv.Så;
import cucumber.api.java.sv.När;

It seems that Eclipse is not recognizing these packages in the repositories. I'm using everything else "as-is" from github (cucumber-jvm 1.1.3).

Please note that I have no problems with:

import cucumber.api.java.sv.Givet;

which makes me suspect something related to the special Swedish characters.

@aslakhellesoy

It seems that Eclipse is not recognizing these packages...

This is contradictory with what you are saying below:

Please note that I have no problems with:
import cucumber.api.java.sv.Givet;

It seems to me that there are no problems with the cucumber.api.java.sv package. Can you confirm that you are talking about classes and not the package?

@hangvar

Yes, you are correct. It appears that the classes (cucumber.api.java.sv.Så and cucumber.api.java.sv.När) is problematic somehow, since there are no errors in Eclipse for import cucumber.api.java.sv.Givet (the import statement is accepted by Eclipse Juno and the annotation @Givet in the step def is also recognized).

@aslakhellesoy

Are your source files saved with UTF-8 encoding?
Can you build from the command line?
Are you getting any error messages?
What's your JDK/OS version?

@hangvar
  • Yes, the source files are saved in UTF-8.

  • No, building gives the following errors from maven:

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[5,27] error: cannot find symbol
[ERROR] symbol: class N▒r
location: package cucumber.api.java.sv
C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[6,27] error: cannot find symbol
[ERROR] symbol: class S▒
location: package cucumber.api.java.sv
C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[17,5] error: cannot find symbol
[ERROR] symbol: class N▒r
location: class HelloStepdefs
C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[22,5] error: cannot find symbol

  • I'm using jdk160_33 and Windows 7.
@aslakhellesoy

Please create a minimal example in a new Github repo that we can clone to reproduce the error.

@hangvar

You find it here:

https://github.com/hangvar/cucumber_helloworld_problem
git://github.com/hangvar/cucumber_helloworld_problem.git

@hangvar

bump

@aslakhellesoy

ack

@aslakhellesoy

I think this might be related to JDK version 6/7 and file name encodings. @dkowis remember that a fes months ago? You wrote ablog post about it too.

I'm not sure, but I think maybe 1.1.3 was the first release done with java 7, after previously using java 6.

@dkowis
Collaborator

Ah yeah, it wasn't actually java, but Java + Apple + Mac Filesystems.

Relevant blog post: http://shlrm.org/blog/2012/10/04/osx-java-utf-8-oh-my/

@hangvar

Hmm, but this shouldn't really be my issue since I'm not on Mac. I run Windows 7.

@aslakhellesoy

It was built on a Mac, which is what might have screwed up the class file names.

@aslakhellesoy

The regression seems to have been introduced in 1.1.2:

curl --silent https://oss.sonatype.org/content/repositories/releases/info/cukes/cucumber-java/1.1.1/cucumber-java-1.1.1.jar | jar tv | awk '{print $8}' | grep cucumber/api/java/sv > cucumber-java-1.1.1-sv.txt
curl --silent https://oss.sonatype.org/content/repositories/releases/info/cukes/cucumber-java/1.1.2/cucumber-java-1.1.2.jar | jar tv | awk '{print $8}' | grep cucumber/api/java/sv > cucumber-java-1.1.2-sv.txt
diff -u cucumber-java-1.1.1-sv.txt cucumber-java-1.1.2-sv.txt

That gives you this diff:

--- cucumber-java-1.1.1-sv.txt  2013-04-04 15:15:32.000000000 +0100
+++ cucumber-java-1.1.2-sv.txt  2013-04-04 15:15:33.000000000 +0100
@@ -1,6 +1,6 @@
 cucumber/api/java/sv/
 cucumber/api/java/sv/Givet.class
 cucumber/api/java/sv/Men.class
-cucumber/api/java/sv/När.class
+cucumber/api/java/sv/När.class
 cucumber/api/java/sv/Och.class
-cucumber/api/java/sv/Så.class
+cucumber/api/java/sv/Så.class

The multibyte lines look the same, but they are actually different. One is using precomposed characters, and the other one is using decomposed. See http://en.wikipedia.org/wiki/Precomposed_character.

Needs further digging to solve, but at least we know the cause.

@hangvar

Hmm, interesting problem. I've tried to look for a unicode converter (NFD to NFC), but I can't find anyone for windows. Also, I'm not sure how that would work on the jar file. I guess building cucumber-jvm locally would render the same problem, unless I can get git to convert to utf-8 NFC. Any other ideas?

@aslakhellesoy

The problem must be solved on my end, not yours. Building and releasing on a Mac with JDK7 currently produces invalid class file names. We have to figure out how to fix the build process.

@dkowis
Collaborator

Release using a Linux box? ;) I could maybe spin up a Rackspace Cloud machine we could use for this.

@os97673
Collaborator

I think Vagrant+VirtualBox would works too ;)

@aslakhellesoy

I need this to work on OS X. I don't want to spin up a VM or use a server to develop.

This is a hard problem to solve. I thought for a long time that the JDK7/OS X compilation problems we were facing earlier could be fixed by writing out DFA converted file names. This did indeed make the JDK7 compiler able to compile the generated annotation classes, but it would produce incorrect class file names.

Then I thought I had fixed the incorrect file name problem by not performing any DFA
conversion at all (like it was before), but to launch javac in a different way. Instead of letting Maven
launch javac (which does it incorrectly on JDK7/OS X), we launch javac ourselves, right after generating the sources.

This made me able to compile non-DFA converted file names on JDK7/OS X, but this created another problem. Stepdefs using the annotations would not compile.

This whole thing is a shit show, and the only reliable fix I can think of is to avoid multibyte characters in file names altogether. I'll send a pull request for review.

@dkowis
Collaborator

@aslakhellesoy Can you format a partition differently? I don't know anything about OSX, but it's the filesystem that's being lame. So if you could use a partition formatted to something else that OSX can understand, perhaps one of the linux filesystems, that would solve the problem, yes?

@dkowis
Collaborator

http://blog.applegrew.com/2011/12/access-ext3ext2-file-system-on-mac-osx-lion-10-7/

Blech, none of these options are particularly good. I'm glad I run linux.

@aslakhellesoy

Thank for looking into this @dkowis

I'm kind of stuck on a Mac, and not planning to get a new machine anytime soon. Any thoughts about my proposed workaround in #487?

@aslakhellesoy

Also check today's #cucumber IRC logs for discussions and links

@os97673
Collaborator

I agree that removing Mac from list of platforms people can use for Cucumber's development is bad idea. So any idea which helps to fix the problem should be considered. And I, personally, like the approach suggested in #487
@aslakhellesoy on the IRC you said

The only annoyance with it is how IntelliJ suggests imports...

what exactly you do not like? Does the IDE suggests something like lang.keyword instead of keyword?

@aslakhellesoy

@os97673 yes. See the comment in #499 for an example of what IDEA does. I'd expect that would be relatively easy to fix in IDEA though.

@os97673
Collaborator

@aslakhellesoy yep, it is a nasty problem, I'd like to see it fixed too ;)

@hangvar

Will this proposed solution be merged and included in the next release?

@aslakhellesoy

Maybe. I'm still having some problems with the proposed solution (only when building on a Mac) so it depends on whether or not I can solve it. Obviously, if anyone can help solving this it would help a lot.

If the problem isn't solveable I'll have to figure out a way to put a Linux inside my mac.

This is a tricky problem to solve.

@dkowis
Collaborator

@aslakhellesoy any news? I don't have a mac, so I can't really help with this.

@aslakhellesoy

We should try to make Travis publish to the staging repo when the version is not a snapshot.
That way the release process would be:

  1. Run the maven goal that removes SNAPSHOT from all poms
  2. Commit and push
  3. Wait for travis to finish
  4. Log in to sonatype and close/publish the artifacts
  5. Tag the release (we'd have a simple script for that that reads the version from the pom)

WDYT?

@dkowis
Collaborator

Hm. Travis is really darn slow. Could we accomplish the same thing with a Vagrant driven VM? That should be faster, since it's an on-demand build, but we don't have to do it this way. We would have to do the work of picking out a suitable VM, and writing a few puppet manifests... ugh, more work.

I'm not a huge fan of this, since it's only Macs and their silly file systems that are affected by this, and I think it's only if the artifacts are built on a Mac? Or did we also figure out that artifacts built on a Linux don't work on a Mac? (I can't remember.)

It does get the job done, however, and I can't really think of anything better

@aslakhellesoy

@dkowis the lead time from a push to a green build is somewhere between 15-30 minutes. I agree it's slow, but in the context of cutting a release every month or so I don't think it matters.

Simplicity trumps speed in this case, and having travis do most of the grunt work for a release would be awesome.

@dkowis
Collaborator

True, if the speed becomes a problem, we'll resolve it later I'm sure.

Ship all the things.

@aslakhellesoy

It would be great to have Travis do the release. It should upload artifacts to the release repo automatically if we remove -SNAPSHOT from all POMs.

One challenge is to sign the artifacts. It might be possible: https://groups.google.com/d/msg/travis-ci/86sOKUoFEy0/vjn8X0xxFzYJ

@dkowis
Collaborator

Given that Travis is sometimes timing out on us, do we still rely on Travis to do solve this problem for us?

@aslakhellesoy

We never relied on it for releases, only snapshots.

I'm back on a Mac with good old trusty java 1.6, which works. And I'm now familiar with vagrant and dockers, so I'm happy to switch to linux if the problem comes back.

Leaving the ticket open though.

@robertoandrade

The fixes on e6beaf4 helped, but the only way I was able to make it actually work on my Mac was by adding -encoding UTF-8 to the javac command line executed by compile-and-delete-generated-i18n-code.sh

@aslakhellesoy

@robertoandrade thanks for the heads up! Fixed in a0e85ec

@aslakhellesoy

This is fixed on master (1.1.6-SNAPSHOT)

@robertoandrade

On my Mac it still fails on master during the maven-jar-plugin execution with a Malformed input or input contains unmappable characters: error:

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ cucumber-java
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:32.707s
[INFO] Finished at: Mon Mar 17 19:12:54 EDT 2014
[INFO] Final Memory: 49M/388M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:2.4:jar (default-jar) on project cucumber-java: Error assembling JAR: java.lang.reflect.InvocationTargetException: Malformed input or input contains unmappable chacraters: <removed-for-brevity>/cucumber-jvm/java/target/classes/cucumber/api/java/ar/????.class -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
@robertoandrade

I finally got to the bottom of it:

  • Apparently it has to do with the terminal (settings) I was using also. Check https://netbeans.org/bugzilla/show_bug.cgi?id=225743.
  • By doing:
    • export LC_CTYPE=UTF-8 before executing maven;
    • Removing your shell scripts altogether;
    • Changing the groovy script on the pom.xml (NFD for NFC): return java.text.Normalizer.normalize(s, java.text.Normalizer.Form.NFC)
    • I was able to use the maven-compiler-plugin as well as maven-jar-plugin to build it successfully.
@robertoandrade

Might also have to do with the fix described in the report I linked to: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8003228

I'm currently running JDK 1.7.0_51 which already contains the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.