Clone Xtext umbrella repository

[kthoms]

Import projects and add them to working set "Releng"

Signed-off-by: Karsten Thoms karsten.thoms@itemis.de

[kthoms]

The change will clone the umbrella repository eclipse/xtext. This will be done for any selected subproject unconditionally. This may be organized also in another subproject if desired. Please comment on that.

The projects in this repository are organized into the working set "Releng".

[cdietrich]

i think we would need this anyway for website and documentation. plus we need a place for the gradle composite build @spoenemann @svenefftinge what do you think

[kthoms]

I would like to clone the repo shallow, but neither JGit nor Oomph support that yet (see bugs 475615, 507324).

[svenefftinge]

That repo is quite huge, why is it needed by contributors?

[cdietrich]

the new gradle does not support workspace composites anymore. => we need a place for a gradle composite. and we need a place for the website. and our releng build is there. were would you put these things otherwise?

[cdietrich]

it looks like we will have a problem importing in oomph as well.
ill have a look how buildship now handles the .classpath files
@oehme can you give us some insights.
how can i import a composite build when not using the gradle wizard nor the ability to execute a refresh gradle project?
would a flat import of all projects work as well?

[cdietrich]

i did some testing and

(1) import composite
(2) import component1
(3) import component2

seems to work

[svenefftinge]

Could you be a bit more elaborate about what you tested? What is composite (0-2) and how did you import?
@oehme I am a bit concerned that the new approach by buildship, to require a composite file is a bit tedious when having different settings. For instance I also have lsp4j in my workspace and would like to have workspace resolution for that. Or if I work on an Xtext language and have xtext-core in my workspace, too. Is there some related discussion thread, resp. documentation of what that 'composite' file should look like and where to put it?

@ALL I could ask for another repository "xtext-umbrella" where we put such common things (also the oomph set up). Than we would at least not need to clone the fat history.

[cdietrich]

we should first make sure this is really needed and it actually works when importing with oomph since oomph does not know anything about gradle

[oehme]

You can make the composite dynamic (use a for-loop to iterate over things checked out next to it for instance). You'll have to import it with Gradle once to get the correct org.eclipse.buildship.core.prefs, after that you can check those in and use oomph. The only constraint is that the "umbrella" project always needs to be checked out, others can be missing.

[cdietrich]

ok so if we once update the prefs files then we can import the projects using the normal "eclipse" project import and buildship will do the magic filling the "Projects and External Dependencies" Classpath Container

[oehme]

@svenefftinge for languages that you are currently developing I'd propose the following:

• have an ignored directory "languages" in the umbrella repo
• check out langugages there
• iterate over that folder in the settings.gradle file of the umbrella project, so they automatically get included in your workspace

We might at some point offer a UI that allows you to say "Hey, treat this as an included build please" without needing to change any file or have a special location. We'd be happy to take a contribution for that :)

[kthoms]

@oehme What do you mean with the "languages" there? It is about all possible projects that reside in the different repositories. So if I have e.g. xtend.core, I would like to have a workspace dependency to xtext.core.

@svenefftinge An umbrella project could help to lower the pain with cloning. It could be "xtext-releng" for example, or "xtext-dev". We can expect that the Oomph/JGit limitation is solved some day, but it would definitely take more time than we want now.

[oehme]

@kthoms there's no difference. All included builds can reference each other. In the simplest case you would check out all your projects into a well known location inside the umbrella directory. That works with git submodules or with tools like MyRepos (and oomph I'd hope) Then all you need is a for-loop in umbrella's settings.gradle to include all those builds into one big one.

[cdietrich]

The forloop is already there in a sample branch I created

[svenefftinge]

We might at some point offer a UI that allows you to say "Hey, treat this as an included build please" without needing to change any file or have a special location. We'd be happy to take a contribution for that :)

If the settings file is entirely generic, couldn't we make oomph generate it (as long as buildship doesn't do it) instead of requiring another repository?

[cdietrich]

well i assume it is the gradle wrapper stuff that needs to be "created" as well. and i dont know if oomph can do that

[kthoms]

Oomph could create the settings file, as a workaround. But I think that Oomph's project import does not do what we need. We have to import the projects as Gradle projects, not as generic projects. I could give it a try next week.
The wrapper stuff could be created with a Gradle launch, I guess.

[cdietrich]

@kthoms id did some experiments and importing as normal projects seem to work (with updated prefs files)

[kthoms]

@cdietrich I can confirm that it seems to work with your composite project.

I also think that this composite file could be created by Oomph.

I think we should handle this topic in a separate issue, not in this PR. The PR still makes sense against the background to import the Releng projects and maybe others later that are placed in the umbrella project. But it should "only" be another subproject.

If agreed, I would update the PR to have another subproject "xtext-devenv" (other name suggestions welcome), which clones eclipse/xtext, imports the projects from there and organizes them in the working set "Releng". Comments?

[oehme]

@kthoms why would you create the composite file with Oomph? That just makes it harder for everyone who doesn't use Oomph/Eclipse and who wants to work on Xtext. As Christian explained, the file is just a generic for-loop which could either be checked into an umbrella repository or into xtext-core (if you don't want an umbrella).

[kthoms]

The issue is that cloning the main repo consumes 500MB and takes time accordingly. And this just for that simple file and relent projects that most users don't need?

[kthoms]

And as mentioned before, shallow clone is not supported by Oomph/JGit yet.

[oehme]

That's why I said you could either have a new umbrella repo or put the composite config in xtext-core.

[kthoms]

It is also valid to not select xtext-core. So that's not the right place.
If there should be a brand new repo I'm fine with that.

[dhuebner]

I think it's a good idea to have a separate umbrella repo. There is currently some releng related code in the eclipse/xtext repo we should also move.

[kthoms]

There were objections against checked in buildship preferences with relative paths pointing to the umbrella project with a relative path. From the discussion I got the following results:

• we have the requirement to resolve cross-repo dependencies on workspace projects
• Buildship is not able to resolve workspace projects without a composite build like m2e or PDE does
• we have to use a milestone of Buildship 2.0 and of Gradle 3.3
• xtext-umbrella contains a composite gradle build
• the build.gradle contains a task generatePrefs and there is a launch config GenerateBuildshipPreferences.launch
• the setup will clone the umbrella repo and import the gradle-composite project for any selection of subprojects
• for each subproject the launch config is executed to generate the buildship preferences after git clone and before project import
• this will connect the projects with the composite build, which is required to resolve cross-repo dependencies
• the org.eclipse.buildship.core.prefs will be deleted from the repositories and ignored
• this will require that projects have to be re-imported as Gradle projects in order to get these settings generated by Buildship when the devenv is not set up by Oomph

[oehme]

for each subproject the launch config is executed to generate the buildship preferences after git clone and before project import

The launch config is in the umbrella and is only run once. It generates the settings for all checked-out Xtext projects.

this will require that projects have to be re-imported as Gradle projects in order to get these settings generated by Buildship when the devenv is not set up by Oomph

This is what you should do with Gradle projects anyway. Checking in .project/.classpath is just a workaround for Oomph's import limitations. If there was a way to run arbitratry importers, we wouldn't be having any of these problems.

[kthoms]

The launch config is in the umbrella and is only run once. It generates the settings for all checked-out Xtext projects.

The launch config must be executed after cloning of the sub-repository/ies. If the launch task would be placed in the main section of the Oomph setup, it would be executed before any subrepo is checked out. Thus as a consequence the launch has to be executed after the clone task of the subrepository. Since it cannot be determined which modules are selected by the user, this has to be placed after each git clone task for each sub repository. This executes the launch multiple times, but does not harm. The launch takes ~10s per execution.

[oehme]

Those 10s can be cut by deactivating the composite configuration with a property @cdietrich. It's not needed to generate the settings.

[kthoms]

I don't get it. Why are the settings now not needed to be generated?
What exactly has to be changed now on the setup?

[cdietrich]

i asume this refers to be generated from the launch config only. and we have to run it for the "last" project we dont know => we run it for every project (dont know a better way in oomph now)

[oehme]

@kthoms the composite configuration is not needed to run your settings generator task. The composite configuration is what takes most of the 10s. I already discussed with @cdietrich that it should be deactivated in settings.gradle when generating the .prefs file.

[cdietrich]

see eclipse/xtext-umbrella@b35a425

[kthoms]

Please review. With the current state the workspace is set up as expected for me, i.e. generating buildship settings and having cross-repo project dependencies.

For example org.eclipse.xtend.core looks like this: