International keywords

[hangvar]

Hi,

I have problems switching the examples to Swedish international keywords. I've tried helloworld as well as java-calculator and both are having problems when I change to the following import statements:

import cucumber.api.java.sv.Så;
import cucumber.api.java.sv.När;

It seems that Eclipse is not recognizing these packages in the repositories. I'm using everything else "as-is" from github (cucumber-jvm 1.1.3).

Please note that I have no problems with:

import cucumber.api.java.sv.Givet;

which makes me suspect something related to the special Swedish characters.

[aslakhellesoy]

It seems that Eclipse is not recognizing these packages...

This is contradictory with what you are saying below:

Please note that I have no problems with:
import cucumber.api.java.sv.Givet;

It seems to me that there are no problems with the cucumber.api.java.sv package. Can you confirm that you are talking about classes and not the package?

[hangvar]

Yes, you are correct. It appears that the classes (cucumber.api.java.sv.Så and cucumber.api.java.sv.När) is problematic somehow, since there are no errors in Eclipse for import cucumber.api.java.sv.Givet (the import statement is accepted by Eclipse Juno and the annotation @givet in the step def is also recognized).

[aslakhellesoy]

Are your source files saved with UTF-8 encoding?
Can you build from the command line?
Are you getting any error messages?
What's your JDK/OS version?

[hangvar]

• Yes, the source files are saved in UTF-8.
• No, building gives the following errors from maven:

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[5,27] error: cannot find symbol
[ERROR] symbol: class N▒r
location: package cucumber.api.java.sv
C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[6,27] error: cannot find symbol
[ERROR] symbol: class S▒
location: package cucumber.api.java.sv
C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[17,5] error: cannot find symbol
[ERROR] symbol: class N▒r
location: class HelloStepdefs
C:\Users\NRKHAN\Downloads\cucumber-jvm-master\cucumber-jvm-master\examples\java-helloworld\src\test\java\cucumber\examples\java\helloworld\HelloStepdefs.java:[22,5] error: cannot find symbol

• I'm using jdk160_33 and Windows 7.

[aslakhellesoy]

Please create a minimal example in a new Github repo that we can clone to reproduce the error.

[hangvar]

You find it here:

https://github.com/hangvar/cucumber_helloworld_problem
git://github.com/hangvar/cucumber_helloworld_problem.git

[hangvar]

bump

[aslakhellesoy]

ack

[aslakhellesoy]

I think this might be related to JDK version 6/7 and file name encodings. @dkowis remember that a fes months ago? You wrote ablog post about it too.

I'm not sure, but I think maybe 1.1.3 was the first release done with java 7, after previously using java 6.

[dkowis]

Ah yeah, it wasn't actually java, but Java + Apple + Mac Filesystems.

Relevant blog post: http://shlrm.org/blog/2012/10/04/osx-java-utf-8-oh-my/

[hangvar]

Hmm, but this shouldn't really be my issue since I'm not on Mac. I run Windows 7.

[aslakhellesoy]

It was built on a Mac, which is what might have screwed up the class file names.

[aslakhellesoy]

The regression seems to have been introduced in 1.1.2:

curl --silent https://oss.sonatype.org/content/repositories/releases/info/cukes/cucumber-java/1.1.1/cucumber-java-1.1.1.jar | jar tv | awk '{print $8}' | grep cucumber/api/java/sv > cucumber-java-1.1.1-sv.txt
curl --silent https://oss.sonatype.org/content/repositories/releases/info/cukes/cucumber-java/1.1.2/cucumber-java-1.1.2.jar | jar tv | awk '{print $8}' | grep cucumber/api/java/sv > cucumber-java-1.1.2-sv.txt
diff -u cucumber-java-1.1.1-sv.txt cucumber-java-1.1.2-sv.txt

That gives you this diff:

--- cucumber-java-1.1.1-sv.txt  2013-04-04 15:15:32.000000000 +0100
+++ cucumber-java-1.1.2-sv.txt  2013-04-04 15:15:33.000000000 +0100
@@ -1,6 +1,6 @@
 cucumber/api/java/sv/
 cucumber/api/java/sv/Givet.class
 cucumber/api/java/sv/Men.class
-cucumber/api/java/sv/När.class
+cucumber/api/java/sv/När.class
 cucumber/api/java/sv/Och.class
-cucumber/api/java/sv/Så.class
+cucumber/api/java/sv/Så.class

The multibyte lines look the same, but they are actually different. One is using precomposed characters, and the other one is using decomposed. See http://en.wikipedia.org/wiki/Precomposed_character.

Needs further digging to solve, but at least we know the cause.

[hangvar]

Hmm, interesting problem. I've tried to look for a unicode converter (NFD to NFC), but I can't find anyone for windows. Also, I'm not sure how that would work on the jar file. I guess building cucumber-jvm locally would render the same problem, unless I can get git to convert to utf-8 NFC. Any other ideas?

[aslakhellesoy]

The problem must be solved on my end, not yours. Building and releasing on a Mac with JDK7 currently produces invalid class file names. We have to figure out how to fix the build process.

[dkowis]

Release using a Linux box? ;) I could maybe spin up a Rackspace Cloud machine we could use for this.

[os97673]

I think Vagrant+VirtualBox would works too ;)

[aslakhellesoy]

I need this to work on OS X. I don't want to spin up a VM or use a server to develop.

This is a hard problem to solve. I thought for a long time that the JDK7/OS X compilation problems we were facing earlier could be fixed by writing out DFA converted file names. This did indeed make the JDK7 compiler able to compile the generated annotation classes, but it would produce incorrect class file names.

Then I thought I had fixed the incorrect file name problem by not performing any DFA
conversion at all (like it was before), but to launch javac in a different way. Instead of letting Maven
launch javac (which does it incorrectly on JDK7/OS X), we launch javac ourselves, right after generating the sources.

This made me able to compile non-DFA converted file names on JDK7/OS X, but this created another problem. Stepdefs using the annotations would not compile.

This whole thing is a shit show, and the only reliable fix I can think of is to avoid multibyte characters in file names altogether. I'll send a pull request for review.

[dkowis]

@aslakhellesoy Can you format a partition differently? I don't know anything about OSX, but it's the filesystem that's being lame. So if you could use a partition formatted to something else that OSX can understand, perhaps one of the linux filesystems, that would solve the problem, yes?

[dkowis]

http://blog.applegrew.com/2011/12/access-ext3ext2-file-system-on-mac-osx-lion-10-7/

Blech, none of these options are particularly good. I'm glad I run linux.

[aslakhellesoy]

Thank for looking into this @dkowis

I'm kind of stuck on a Mac, and not planning to get a new machine anytime soon. Any thoughts about my proposed workaround in #487?

[aslakhellesoy]

Also check today's #cucumber IRC logs for discussions and links

[os97673]

I agree that removing Mac from list of platforms people can use for Cucumber's development is bad idea. So any idea which helps to fix the problem should be considered. And I, personally, like the approach suggested in #487
@aslakhellesoy on the IRC you said

The only annoyance with it is how IntelliJ suggests imports...

what exactly you do not like? Does the IDE suggests something like lang.keyword instead of keyword?

[aslakhellesoy]

@os97673 yes. See the comment in #499 for an example of what IDEA does. I'd expect that would be relatively easy to fix in IDEA though.

[os97673]

@aslakhellesoy yep, it is a nasty problem, I'd like to see it fixed too ;)

[hangvar]

Will this proposed solution be merged and included in the next release?

[aslakhellesoy]

Maybe. I'm still having some problems with the proposed solution (only when building on a Mac) so it depends on whether or not I can solve it. Obviously, if anyone can help solving this it would help a lot.

If the problem isn't solveable I'll have to figure out a way to put a Linux inside my mac.

This is a tricky problem to solve.

[dkowis]

@aslakhellesoy any news? I don't have a mac, so I can't really help with this.

[aslakhellesoy]

We should try to make Travis publish to the staging repo when the version is not a snapshot.
That way the release process would be:

1. Run the maven goal that removes SNAPSHOT from all poms
2. Commit and push
3. Wait for travis to finish
4. Log in to sonatype and close/publish the artifacts
5. Tag the release (we'd have a simple script for that that reads the version from the pom)

WDYT?

[dkowis]

Hm. Travis is really darn slow. Could we accomplish the same thing with a Vagrant driven VM? That should be faster, since it's an on-demand build, but we don't have to do it this way. We would have to do the work of picking out a suitable VM, and writing a few puppet manifests... ugh, more work.

I'm not a huge fan of this, since it's only Macs and their silly file systems that are affected by this, and I think it's only if the artifacts are built on a Mac? Or did we also figure out that artifacts built on a Linux don't work on a Mac? (I can't remember.)

It does get the job done, however, and I can't really think of anything better

[aslakhellesoy]

@dkowis the lead time from a push to a green build is somewhere between 15-30 minutes. I agree it's slow, but in the context of cutting a release every month or so I don't think it matters.

Simplicity trumps speed in this case, and having travis do most of the grunt work for a release would be awesome.

[dkowis]

True, if the speed becomes a problem, we'll resolve it later I'm sure.

Ship all the things.

[aslakhellesoy]

It would be great to have Travis do the release. It should upload artifacts to the release repo automatically if we remove -SNAPSHOT from all POMs.

One challenge is to sign the artifacts. It might be possible: https://groups.google.com/d/msg/travis-ci/86sOKUoFEy0/vjn8X0xxFzYJ

[dkowis]

Given that Travis is sometimes timing out on us, do we still rely on Travis to do solve this problem for us?

[aslakhellesoy]

We never relied on it for releases, only snapshots.

I'm back on a Mac with good old trusty java 1.6, which works. And I'm now familiar with vagrant and dockers, so I'm happy to switch to linux if the problem comes back.

Leaving the ticket open though.

[robertoandrade]

The fixes on e6beaf4 helped, but the only way I was able to make it actually work on my Mac was by adding -encoding UTF-8 to the javac command line executed by compile-and-delete-generated-i18n-code.sh

[aslakhellesoy]

@robertoandrade thanks for the heads up! Fixed in a0e85ec

[aslakhellesoy]

This is fixed on master (1.1.6-SNAPSHOT)

[robertoandrade]

On my Mac it still fails on master during the maven-jar-plugin execution with a Malformed input or input contains unmappable characters: error:

[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ cucumber-java
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:32.707s
[INFO] Finished at: Mon Mar 17 19:12:54 EDT 2014
[INFO] Final Memory: 49M/388M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:2.4:jar (default-jar) on project cucumber-java: Error assembling JAR: java.lang.reflect.InvocationTargetException: Malformed input or input contains unmappable chacraters: <removed-for-brevity>/cucumber-jvm/java/target/classes/cucumber/api/java/ar/????.class -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[robertoandrade]

I finally got to the bottom of it:

• Apparently it has to do with the terminal (settings) I was using also. Check https://netbeans.org/bugzilla/show_bug.cgi?id=225743.
• By doing:
  • export LC_CTYPE=UTF-8 before executing maven;
  • Removing your shell scripts altogether;
  • Changing the groovy script on the pom.xml (NFD for NFC): return java.text.Normalizer.normalize(s, java.text.Normalizer.Form.NFC)
  • I was able to use the maven-compiler-plugin as well as maven-jar-plugin to build it successfully.

[robertoandrade]

Might also have to do with the fix described in the report I linked to: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8003228

I'm currently running JDK 1.7.0_51 which already contains the fix.

[jeeveshmishra]

Hi Guys ,

Has this issue been fixed? We run our system on JDK7, Windows 10, Cucumber feature file uses Norwegian language.

We are getting following compilation errors:
Error:(8, 28) java: cannot find symbol
symbol: class NÃ¥r
location: package cucumber.api.java.no
Error:(9, 28) java: cannot find symbol
symbol: class SÃ¥
location: package cucumber.api.java.no
Error:(41, 6) java: cannot find symbol
symbol: class NÃ¥r
location: class com.yabank.cucumber.feature.CreditCardApplicationScenario
Error:(47, 6) java: cannot find symbol
symbol: class NÃ¥r
location: class com.yabank.cucumber.feature.CreditCardApplicationScenario
Error:(53, 6) java: cannot find symbol
symbol: class SÃ¥
location: class com.yabank.cucumber.feature.CreditCardApplicationScenario

Kindly help.

Thanks.

[mpkorstanje]

@jeeveshmishra which version of cucumber are you using?

Also please ensure your file encoding is utf-8.

[jeeveshmishra]

Hi we are using 2.5.1 version of cucumber.

…

On Sep 8, 2017 17:05, "M.P. Korstanje" ***@***.***> wrote: @jeeveshmishra <https://github.com/jeeveshmishra> which version of cucumber are you using? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#487 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALbBaxpSQY-NU0wOYL9hy9fHWZE19Tdkks5sgVejgaJpZM4AgqE5> .

[brasmusson]

Norwegian feature files works on Windows 10 (and Java8), both with Cucumber-JVM 1.2.5 and with Cucumber-JVM 2.0.0 (I did just test that) - there is no version 2.5.1. It is however important to save the files using utf-8 (and utf-8 is not the default file format on Windows).

[jeeveshmishra]

Hi, My apologies for the wrong version number as I did not have my machine. And you are right in the version no 1.2.5. Though we have jdk 1.7. Can there be a workaround for jdk 1.7? Regards, Jeevesh. +47 93983073.

…

On Sep 8, 2017 22:36, "Björn Rasmusson" ***@***.***> wrote: Norwegian feature files works on Windows 10 (and Java8), both with Cucumber-JVM 1.2.5 and with Cucumber-JVM 2.0.0 (I did just test that) - there is no version 2.5.1. It is however important to save the files using utf-8 (and utf-8 is not the default file format on Windows). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#487 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALbBa7NiZcdR4sEBW2WY0-TtTl_ZIhTKks5sgaU7gaJpZM4AgqE5> .

[aslakhellesoy]

What file encoding are the .feature files saved as? It needs to be UTF-8.
What editor are you using?

File encoding needs to be UTF-8.

[jeeveshmishra]

Please see the following attachments.

[aslakhellesoy]

In order to investigate this further, please create a git repo with one feature file and a build script (pom.xml) that will reproduce the problem.

Then, create a new ticket with a link to your repo. Nobody is monitoring 4 year old closed tickets.

[brasmusson]

@jeeveshmishra Your screen shot explicitly says Information: java: Multiple encodings set for module chunk comyabankcucumberpoc "ISO-8859-1" will be used by compiler, you should fix that so the compiler uses UTF-8.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.