Regex for extracting xtext version fails on actual published versions.

[NicolasRouquette]

I've been stumped by a weird gradle dependency resolution failure:

* What went wrong:
Could not resolve all dependencies for configuration ':gov.nasa.jpl.imce.oml.dsl.ui:detachedConfiguration2'.
> Could not resolve org.eclipse.xtend:org.eclipse.xtend.core:1233.
  Required by:
      project :gov.nasa.jpl.imce.oml.dsl.ui
   > Could not resolve org.eclipse.xtend:org.eclipse.xtend.core:1233.
      > Could not get resource 'http://download.eclipse.org/eclipse/updates/4.6/R-4.6.3-201703010400/plugins/ivy_1233.xml'.
         > Could not GET 'http://download.eclipse.org/eclipse/updates/4.6/R-4.6.3-201703010400/plugins/ivy_1233.xml'.
            > Connect to download.eclipse.org:80 [download.eclipse.org/198.41.30.199] failed: Operation timed out (Connection timed out)

Looking at XtendLanguageBasePlugin.automaticallyInferXtendCompilerClasspath which uses a regex version pattern matcher, I see that the problem comes from that regex:

static val LIB_PATTERN = Pattern.compile("org\\.eclipse\\.xtext\\..*-(\\d.*?).jar")

When applied to "org.eclipse.xtext:org.eclipse.xtext.ui:2.11.1.v20170201-1233", it binds to the suffix, "1233".

For more details about this problem, see: http://stackoverflow.com/questions/43359398/gradle-fails-to-resolve-a-weird-dependency-from-a-detached-configuration

[cdietrich]

@oehme any news on this. there are more and more people stumbling on this
https://www.eclipse.org/forums/index.php/t/1086605/

[oehme]

I don't have time to look into this, could you fix the regex and add some unit tests?

[cdietrich]

is there an option to specify the version explictely?

[oehme]

Sure, but that's inconvenient for the user. Instead, let's just fix the regex.

[cdietrich]

i will have a look if the mwe regex fix we recently did might be applicable here as well

[oehme]

The workaround is setting xtext.version explicitly.

[cdietrich]

do we have some list of what we expect for the version extraction???

		assertEquals("2.12", underTest.getXtextVersion(new TestFileCollection(#{
			new File("org.eclipse.xtext.xbase.lib-2.12.jar")
		})))
		assertEquals("2.12.0", underTest.getXtextVersion(new TestFileCollection(#{
			new File("org.eclipse.xtext.xbase.lib-2.12.0.jar")
		})))
		assertEquals("2.12.0.12345", underTest.getXtextVersion(new TestFileCollection(#{
			new File("org.eclipse.xtext.xbase.lib-2.12.0-12345.jar")
		})))
		assertEquals("2.12.0", underTest.getXtextVersion(new TestFileCollection(#{
			new File("org.eclipse.xtext.xbase.lib-2.12.0-v12345.jar")
		})))
		assertEquals("2.12.0", underTest.getXtextVersion(new TestFileCollection(#{
			new File("org.eclipse.xtext.xbase.lib-2.12.0.v12345-6789.jar")
		})))

[oehme]

Is Xtext published with different versioning schemes for different artifacts now? I assumed that the version of ui and core should be exactly the same.

[cdietrich]

Don't know where these strange schemes come from. It seems there is Xtext ui involved in both cases. But don't know where they've got it from

I thought we don't publish them since 2.11

[cdietrich]

Looks like here the p2 bundles are consumes https://github.com/JPL-IMCE/gov.nasa.jpl.imce.oml.core/blob/master/build.gradle

[NicolasRouquette]

@cdietrich FYI: we used the Mavenized repos that that Xtext CI builds to resolve Xtext dependencies. It's a fragile hack; for details, see:

https://github.com/JPL-IMCE/gov.nasa.jpl.imce.oml.core/blob/master/build.gradle#L68
https://github.com/JPL-IMCE/gov.nasa.jpl.imce.oml.core/blob/master/build.gradle#L76
https://github.com/JPL-IMCE/gov.nasa.jpl.imce.oml.core/blob/master/build.gradle#L83
https://github.com/JPL-IMCE/gov.nasa.jpl.imce.oml.core/blob/master/build.gradle#L95
https://github.com/JPL-IMCE/gov.nasa.jpl.imce.oml.core/blob/master/build.gradle#L103

We switched to Tycho because there is better support for resolving against P2 and/or Maven repos; however, it isn't a whole lot better either.

It would be a lot easier if everything about Xtext were published properly to both P2 & Maven repos. That way, we could use Xtext without having to use P2-specific workarounds.

[cdietrich]

Xtext P2 Artifacts indeed have versioning like 2.13.0.v20170601-1224

[smelc]

We're facing the same issue and have made the same observation as @NicolasRouquette. We've been using gradle to compile our command line compilers, because we lack experience with tycho; so we have to use p2 workarounds (such as p2asmaven).

[cdietrich]

Does the workaround mentioned above work?

[smelc]

I'm in a slightly different context than the reports above since this bug affects one of my own variable which ends up being inspected by the xtext gradle plugin because it appears in a mwe2 task (as in this example: https://blogs.itemis.com/en/using-xtext-with-xcore-and-gradle). But yes, changing the variable content from "2.13.0.v20171020-0732" (version published by xtext for 2.13) to "2.13.0" did workaround this bug. But then gradle fails at downloading the dependencies, which I believe was mentioned elsewhere already: https://www.eclipse.org/forums/index.php?t=msg&th=1084098&goto=1753808&#msg_1753808 (2.13.0 vs 2.13.0.vFOO mismatch).

So in the end we've ended up specifying the jars explicitly like this:

compile files(file(".").absolutePath + "build/p2asmaven/p2runnable/" + "org.eclipse.xtext/plugins/org.eclipse.xtext.builder_2.13.0.v20171020-0732.jar")

i.e. we're just using gradle to download the p2 repos automatically (thanks to p2asmaven), but xtext dependency management cannot be handled gracefully.

[cdietrich]

i dont get that. the plugin stuff (finding out the xtext version and download it) and having xtext.builder somethere (which is an eclipse plugin) are different things arent they?

=> can you be more explicit that you are doing?

[cdietrich]

btw pull requests are welcome, but i cannot tell you how to fix this without breaking anything

[ghost]

(@smelc speaking) Our gradle script is roughly as follows (see "xtextVersion" definition and its usages below).

subprojects {
    repositories {
       mavenCentral()
       jcenter()
       maven { url 'https://repo.eclipse.org/content/groups/releases/' } // Eclipse repo
       maven { url 'https://maven.repository.redhat.com/ga/' } // GA repo
       maven { url 'http://central.maven.org/maven2/' } // Maven central
       maven { url 'http://repo1.maven.org/maven2/' } // Maven repo1
    }

    ext {
        xtextVersion = '2.13.0' // '2.13.0.v20171020-0920'
        xtextGradlePluginVersion = '1.0.19'
        xpandVersion = '2.0.0' // '2.2.0.v201605260315' doesn't work // '2.2.0' doesn't work
    ...
    }
}

dependencies {
    compile "org.eclipse.xtext:org.eclipse.xtext:${xtextVersion}"
    compile "org.eclipse.xtext:org.eclipse.xtext.xbase:${xtextVersion}"
}

configurations {
    mwe2 {
        extendsFrom compile
    }
}

dependencies {
    mwe2 "org.eclipse.xtext:org.eclipse.xtext.common.types:${xtextVersion}"
    mwe2 "org.eclipse.xtext:org.eclipse.xtext.xtext.generator:${xtextVersion}"
    mwe2 "org.eclipse.xtext:xtext-antlr-generator:[2.1.1, 3)"

    ...
}

But unfortunately there's no way to get it working right. If we set "xtextVersion" to 2.13 or to "2.13.0.v20171020-0920", it doesn't work. With the latter it fails with the bug detailed in this ticket; while with 2.13 it fails at finding some jars (org.eclipse.xtext.builder being the most prominent one, but there should be others).

Note that we're not gradle experts, so maybe we're not doing things right; but it was working well with xtext 2.9 and things broke when updating to 2.13 (mainly because 2.9 is better published on maven centtral). Hence we started using p2asmaven to try to get things working (so that our nightly regression tests become available again). We know that our dependencies extend a bit too far in the UI (org.eclipse.xtext.builder being an example), but it's part an historical issue (we're working on it for this one) and part intended (our command line tests include quickfixes, for which we emulate part of Eclipse UI).

Surely something that would help would be to have well defined maven-consumable xtext repos.

[cdietrich]

i dont see where you set xtext version explicitely to the gradle plugin

[ghost]

@cdietrich> as I said we don't use the xtext version from the xtext plugin (as the recipe from https://blogs.itemis.com/en/using-xtext-with-xcore-and-gradle doesn't apply since we don't use xcore, but simply generates our classes out of an .xtext grammar). If you're talking about the version of the gradle xtext plugin (1.0.19), it's set in the snippet below. As I've said already, we're not gradle experts; so if there's a reference way to do these things, I'll be happy to try it.

buildscript { // 'buildscript' means dependencies of the script itself, not the project being built
    // it cannot be shared in xtext-source-layout.gradle :-(

    repositories {
        jcenter()
    }

    dependencies {
        classpath "org.xtext:xtext-gradle-plugin:${xtextGradlePluginVersion}"
    }
}

apply plugin: 'org.xtext.xtend'
apply from: file(".").absolutePath + "/../gradle/xtext-source-layout.gradle"

dependencies {
    compile project(':com.provenrun.opensource')
    compile project(':com.provenrun.shared')
    compile project(':com.provenrun.shared.xtext')

    compile files(file(".").absolutePath + p2_path + "org.eclipse.sdk/plugins/org.eclipse.core.runtime_3.13.0.v20170207-1030.jar")
}

task generateXtextLanguage(type: JavaExec) {
	main = 'org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher'
	classpath = configurations.mwe2
	classpath 'src' // I'd like to reuse main.java.srcDirs instead
	inputs.file 'src/com/provenrun/smdt/smil/GenerateSMIL.mwe2'
	inputs.file 'src/com/provenrun/smdt/smil/SMIL.xtext'
	outputs.dir 'src-gen'
	args += 'src/com/provenrun/smdt/smil/GenerateSMIL.mwe2'
	args += '-p'
	args += "rootPath=/${projectDir}/.."
}

generateXtext.dependsOn(generateXtextLanguage)
clean.dependsOn(cleanGenerateXtextLanguage)

classes.dependsOn(generateXtextLanguage)
cleanClasses.dependsOn(cleanGenerateXtextLanguage)

[cdietrich]

i simply mean: if you set it explicitely you can use for xtext dependencies you use whatever you like

[ghost]

@cdietrich > oh I get your comment now. Indeed I wasn't setting xtext's version. I had in mind that the version was read from the global variable "xtextVersion", but it's "xtext.version". Setting it indeed workarounds this bug which I was hitting again in a different project.