When a devfile is included in a GitHub repo the project to clone should be inferred

[maxandersen]

If I have project without a devfile.yaml and links to it like https://che.openshift.io/f?url=https://github.com/maxandersen/quarkus-quickstarts/tree/che a empty workspace is setup but with the project checked out.

If I add a devfile.yaml the project is not checked out and so far I've only been able to get something imported by adding:

projects:
  -
    name: quarkus-quickstarts
    source:
      type: git
      location: "https://github.com/maxandersen/quarkus-quickstarts.git"

but that now means I'm checking out a hardcoded path and thus the devfile.yaml inside a project is not really working well as I'm getting setup done from a branch potentially a completely different location/project and project checked out in che is from this hardcoded place.

@gorkem suggested to try:

projects:
  -
    name: quarkus-quickstarts
    source:
      type: git
``

but that gave me:  
"Error occurred during creation a workspace from devfile located at https://raw.githubusercontent.com/maxandersen/quarkus-quickstarts/che/devfile.yaml. Cause: Devfile schema validation failed. Error: (/projects/0/source):The object must have a property whose name is "location".""

Thus I can't see how to make a `devfile.yaml` that is easy to maintain as I would literally have to update that reference on every tag/branch I make for my project.

[ibuziuk]

Looks like an important enhancement - if the project is not specified explicitly in devfile, project should be cloned based on the url attribute

[maxandersen]

Looks like an important enhancement - if the project is not specified explicitly in devfile, project should be cloned based on the url attribute

+1, would be great if the location would be relative to the cloned url.

i.e.

location = new URL(clonedurl, locationstr)

In case clonedurl is https://github.com/quarkus/quarkus-quickstarts:

path	result
	https://github.com/quarkus/quarkus-quickstarts
../quarkusio	https://github.com/quarkus/quarkusio
https://github.com/maxandersen/xam.dk	https://github.com/maxandersen/xam.dk

Allows you to clone a collection git repo under one base which matches to an organization on github and more.

[skabashnyuk]

@maxandersen I'm sorry I can't get the idea of this task. Can you give some example of devfiles that is not working and what you want to make working?

[maxandersen]

Take https://github.com/maxandersen/quarkus-quickstarts/tree/che has a devfile.yaml as an example - it has a devfile.yaml that currently will checkout itself but not import any projects.

When I add a project section I am currently required to put a hardcoded url to a git repo, like https://github.com/maxandersen/quarkus-quickstarts/ - that means it only ever will work for my github repo; i.e. you won't actually be possible to fork it and then work on your own variation. Also if I move github location I am forced to update this devfile.yaml which is not something you'll remember to do always :)

Thus my suggestion is to either allow an empty location to indicate it is the same from where the devfile.yaml was found or expose a property or enviroment variable one can use in the location field so it does not become hardcoded.

[skabashnyuk]

Thus my suggestion is to either allow an empty location to indicate it is the same from where the devfile.yaml was found or expose a property or enviroment variable one can use in the location field so it does not become hardcoded.

ok for me. CC @l0rd

[l0rd]

We should automatically add the project in the workspace definition. Even if there is no projects section in the devfile. I don't think we should allow a project without a location url.

[maxandersen]

well, if my devfile is from one of the standalone devfile's then it doesn't make sense to checkout the surrounding project.

[skabashnyuk]

@l0rd

We should automatically add the project in the workspace definition.

So devfile in a git repo may not contain location?

Even if there is no projects section in the devfile. I don't think we should allow a project without a location url.

In a workspace.devfile - location is a mandatory thing

[maxandersen]

In a workspace.devfile - location is a mandatory thing

yes, which is the root issue of this - I don't want to put a hardcoded git location for checkout.

[skabashnyuk]

yes, which is the root issue of this - I don't want to put a hardcoded git location for checkout.

@maxandersen workspace.devfile without location means workspace without sources. Are you sure you want exactly that?

[maxandersen]

No, I want workspaces with sources, but I don't want to hardcode my git location as then when the project that has the devfile is forked the user will be checking out the wrong repo, possibly outdated and might not even exist anymore.

ie. I 'm trying to have a devfile inside my project at https://github.com/maxandersen/quarkus-quickstarts and be able to tell others to fork my project and use it in on che.openshift.io and then have it "just work" without pointing to my repo.

Does that help ?

[Mbd06b]

Would a good analogy be like the Jenkinsfile 'scm checkout' declaration?

'scm checkout' is a directive that tells Jenkins to find the Jenkinsfile defined in source repository (git or svn), and build the CI pipeline declared there (usually located in the root directory). This allows the CI pipeline to be source controlled within the project itself, and enables the configuration to be collaboratively amended by project contributors.

[benoitf]

it seems to be important for the usecase where someone is forking a repository or creating a branch for a given repository.

Devfile is automatically referencing the correct location with correct branch.

[gazarenkov]

Why do we need process a devfile storing in a project tree?
It clearly creates so many difficulties and confuse, but what is the benefit other than demo this feature in a ideal conditions?

[gorkem]

If a devfile is referenced directly then projects is applied directly. If a devfile is being processed as part of reference to a git tree then projects che should skip processing the current git repo and process only other repos (if any) referenced from projects section.

[mshaposhnik]

Just to make sure i understood it right:

• Direct devfile reference (i.e. f?url=......./devfile.yaml) - process what in the projects section;
• Git repo reference, no projects section inside devfile - process current repo and branch/tag;
• Git repo reference, devfile contains projects sections - process what is written in it;

Correct ? @maxandersen @gorkem

[gazarenkov]

To make it clear, it is "GitHub repo", not a generic "Git repo", indeed (please correct me if I am wrong) Repo reference works something like this:

• Che checks if it is github.com URL (e g https://github.com/eclipse/che) + branch (e g 7.7.x) and if so
• tries to get the content from something like https://raw.githubusercontent.com/eclipse/che/7.7.x/devfile.yaml

To make a bit clear the question I asked above:
Do we really need to put that much effort trying to solve all of these conflicts and supporting additional code? What's the benefit?
Is not it simpler and more intuitive (if we really need to store devfile in ANY github repo/branch/commit) to remove it and put direct raw.githubusercontent URL instead?

[maxandersen]

• Git repo reference, no projects section inside devfile - process current repo and branch/tag;

I think that is what one is expected - or at least in the projects section be able to refer to the "git url" the project is created from as a way to avoid the hardcoding.

[maxandersen]

@gazarenkov I don't really follow what you are suggesting. The current model seem to just not work.

How are you expecting people to start using Che when the primary configuration file che uses is hardcoded to exact specific git locations ?

It feels like the current devfile approach is from a usecase where it is expected users will have a devfile to start with that is specific to a single repo rather than the more common usecase - users have no devfile or want to be able to have devfile in the repo so it can be shared.

Your suggestion about putting a raw githubuser content url brings what value ? if that file still hardcodes to the github repo for eclipse/che/7.7.x then the project is still configured to be something I can't use/push to unless i'm a eclipse/che committer.

[benoitf]

@mshaposhnik I would say so

[gazarenkov]

@maxandersen

I don't really follow what you are suggesting. The current model seem to just not work.

I mean the current model is not working in certain cases because of unclear expectations we have to devfile stored inside project tree.

I am proposing to completely decouple devfile storage from project files storage because it makes confusions like chicken/eggs problem like: "to see/manage projects sources you need to run a workspace configuration stored together with this project sources" (and btw, Che workspace may contain more than one project). Also (or that's why) it creates a lot of implementation issues we are struggling with for a long time.

How are you expecting people to start using Che when the primary configuration file che uses is hardcoded to exact specific git locations ?

I would expect people to start using Che either:
1/ by choosing from predefined set of devfiles from user dashboard (beginner case)
2/ providing devfile URL to user dashboard
3/ uploading devfile content to user dashboard
4/ providing devfile URL to factory URL

[gorkem]

@mshaposhnik

Git repo reference, devfile contains projects sections - process what is written in it;

I think in this case, current repo reference should also override the conflicting parts of the projects section.

[maxandersen]

I am proposing to completely decouple devfile storage from project files storage because it makes confusions like chicken/eggs problem like: "to see/manage projects sources you need to run a workspace configuration stored together with this project sources" (and btw, Che workspace may contain more than one project). Also (or that's why) it creates a lot of implementation issues we are struggling with for a long time.

I fully agree that devfile have issues for a long time and that devfile always seem to have very concrete hard bindings rather than decoupled declarative approach makes it really hard to reuse/share and the content gets very verbose very quickly.

I would expect people to start using Che either:
1/ by choosing from predefined set of devfiles from user dashboard (beginner case)
2/ providing devfile URL to user dashboard
3/ uploading devfile content to user dashboard
4/ providing devfile URL to factory URL

all of those are fine at the time you actually have decided to use Che - but that’s not where majority of users trying to use an IDE for their project starts. They have an existing project or wantiing to try it - why should users have to know which of potentially many devfiles choose ? why should maintainers of a source repo have to put separate devfiles outside this repo to import it ? how will these projects be possible to share without also having to share the devfile ?

any other IDE/editor are able to store metadata in some form that IDE's then use to open the project without hardcoding source code cloning location /usernames/etc ?

[gazarenkov]

@maxandersen I think the main difference we have is that you consider a workspace configuration (a devfile) as a part of particular project but in Che it is designed just opposite - a project is a part of workspace configuration.
For your questions:

They have an existing project or wantiing to try it - why should users have to know which of potentially many devfiles choose ?

Well, by analogy - when user choose a docker image to run the project it looks like fair to expect him(her) to approximately know something about software installed there. It is pretty the same - user suppose to know that it is a Java (for example) project and can choose devfile to put his project in (btw for this purpose I'd consider having devfiles to be used as a template w/o projects) or plugin to be put in his/her custom devfile. Of course in a case it is clear what inside, i.e. it is well titled/described :)

why should maintainers of a source repo have to put separate devfiles outside this repo to import it ?

I can not see any problems there as soon as workspace config (a devfile) is considered as a first class citizen and not a part of the project. It is the price of having declarative environment (most probably prepared for you by someone else) and not configure runtime env manually like in local IDEs

how will these projects be possible to share without also having to share the devfile ?

Sharing the devfile is all needed since it contains a reference to the project(s). And, yes, if user wants to fork the repo it means this workspace config is changed - i.e devfile is not the same, project location changed.

any other IDE/editor are able to store metadata in some form that IDE's then use to open the project without hardcoding source code cloning location /usernames/etc ?

yes, depends on metadata. If it is related to the runtime for dev env (IDE) itself which Che should know before cloning the project - then storing it inside project tree confusing me the way I described above.

[maxandersen]

I can not see any problems there as soon as workspace config (a devfile) is considered as a first class citizen and not a part of the project. It is the price of having declarative environment (most probably prepared for you by someone else) and not configure runtime env manually like in local IDEs

That is just completely false IMO. You do not need to upfront have such config; it should be possible to autodetect and be derived from your project contents and not be hardcoded upfront. hardcoding should be a choice not a requirement.

Also "most probably prepared for you by someone else"

• a) majority of users coming to Che will not have that and I can tell you the pre-defined we have now are so static and outofdate it prooves it is not scaling as it is now
  b) I here are describing exactly the case where I 'm trying to setup a project or if I absolutely must a devfile that I can give to someone else for them to share...the format of the devfile prevents that by assuming the people I share with will always use my fork.

If a devfiles continues to be only for tons of hardcoded references then I believe che needs something else/new to become useful.

It is absolutely wrong to enforce on users to have to figure out the detailed docker/tooling environment up front and IF you really wanted to that it is killing uptake that these configurations are not even shareable or in anyway possible to parameterise to it can be used inside and outside teams.

I can go and use Eclipse, vscode, netbeans, gitpod.io, vscode online and a bunch of other IDE's instantly with projects today and none of them needs this kind tooling specific setup to get started but ALL of them have ways to after the fact customize, store, share configurations etc.

Che is the one that is standing out making it extremely hard to use because it insists on its configuration to be defined upfront and not be shareable. If the devfile is not way to do that then I suggest we find something else to use to allow easy and shareable confgurations.

[maxandersen]

how will these projects be possible to share without also having to share the devfile ?

Sharing the devfile is all needed since it contains a reference to the project(s). And, yes, if user wants to fork the repo it means this workspace config is changed - i.e devfile is not the same, project location changed.

what you are saying is that devfile is too limited to be used for shared config. something IDE's historically does pretty well but Che doesn't then. I think that is a very big problem and something we should address.

[maxandersen]

yes, depends on metadata. If it is related to the runtime for dev env (IDE) itself which Che should know before cloning the project - then storing it inside project tree confusing me the way I described above.

this is just not right - I fully get che is trying to offer the feature of having pre-defined workspaces - but then we need to get that separated out or at least made possible to parameterize or deduce more dynamically until the user actually want to lock down. The getting started experience is just not good.

[metlos]

I think that we're all here in a kind of strange violent agreement. I think we all have the same end goal - have a smooth experience with an online IDE. So far, we've approached it from a different angle than the one of having a super smooth experience on github.

On a more practical front, I think we've broadened the discussion on this issue way beyond the initial scope. Let's try to take away some inputs and put them in the backlog (if they're not there already) ;)

That is just completely false IMO. You do not need to upfront have such config; it should be possible to autodetect and be derived from your project contents and not be hardcoded upfront. hardcoding should be a choice not a requirement.

Agreed that the experience should be smoother. We're just not there yet. But autodetection and derivation makes the workspace essentially not reproducible - 2 users at two different points in time may end up with a different workspace for the same project.

Also "most probably prepared for you by someone else"

a) majority of users coming to Che will not have that and I can tell you the pre-defined we have now are so static and outofdate it prooves it is not scaling as it is now

I personally don't see it this way. I may be wrong here, but the use case of having a clickable button on github is a nice thing to have but it is not the main use case of Che, which I personally see more behind a corporate firewall - or rather with projects that strongly favor reproducibility and consistency in developer workspaces.

The pre-defined devfiles being out of date actually means that they are reproducible :) Yes, we could make a better job at keeping our stacks (i.e. ready to use initial project templates) up-to-date with the upstream but we need to do that in a versioned way.

b) I here are describing exactly the case where I 'm trying to setup a project or if I absolutely must a devfile that I can give to someone else for them to share...the format of the devfile prevents that by assuming the people I share with will always use my fork.

I think we've made at least some progress with #15765 which enables you to use a project-less devfile in your repo and deducing the project in the factory upon import (e.g. you point Che to your github repo and it will use it as the project if there are no other in the devfile).

If a devfiles continues to be only for tons of hardcoded references then I believe che needs something else/new to become useful.

I think the reproducibility of the workspace environment is actually a feature, not a bug.

It is absolutely wrong to enforce on users to have to figure out the detailed docker/tooling environment up front and IF you really wanted to that it is killing uptake that these configurations are not even shareable or in anyway possible to parameterise to it can be used inside and outside teams.

We could and should do a better job at helping people author the devfile and I think autodetection is a big part of that. We just took the easier path to having a featureful workspace description and that is "hardcoding" the workspace environment. On the other hand we do prepare a rather large set of predefined "stacks" which people can use to kickstart their development and fine-tune their devfiles based on their needs.

I do agree with you that we should have a much better experience when initiating a project from existing sources without a devfile. Currently, we're trying to solve that by having the stacks that you base your initial project on and add your sources. Maybe we should start thinking about adding autodection to the picture. But as you very well know, autodetection is hard and never perfect.

I can go and use Eclipse, vscode, netbeans, gitpod.io, vscode online and a bunch of other IDE's instantly with projects today and none of them needs this kind tooling specific setup to get started but ALL of them have ways to after the fact customize, store, share configurations etc.

In my experience with IDEs that is not true. I am clearly not able to install stock Eclipse and start to work on say a Node.js application without any additional setup. Eclipse may provide syntax highlighting out of the box, but I am pretty sure it doesn't understand the project setup without any additional plugins. I am also sure that my Eclipse installation is going to be different from my colleagues'. But devfile doesn't deal with just the IDE which we've been concentrating on here. I can also define the database server that my application needs, or in a microservice world - I can declare the cooperating microservices that I am not working on directly as just components running within the workspace and I don't need to deal with setting them up or building them at all. IMHO no IDE does that as of yet and no autodetection can do that either.

Che is the one that is standing out making it extremely hard to use because it insists on its configuration to be defined upfront and not be shareable. If the devfile is not way to do that then I suggest we find something else to use to allow easy and shareable confgurations.

We were bad at sharability using forks (and note that behind a corporate firewall, you don't use forks all that much in my humble experience) and based on your great input, I think we've improved the situation.

[skabashnyuk]

Done as part of this eclipse-che/che-docs#1051 and this #15765

[maxandersen]

first of - thanks very much for good feedback here and the #15765 that makes this somewhat easier to work with.

A few comments:

That is just completely false IMO. You do not need to upfront have such config; it should be possible to autodetect and be derived from your project contents and not be hardcoded upfront. hardcoding should be a choice not a requirement.

Agreed that the experience should be smoother. We're just not there yet. But autodetection and derivation makes the workspace essentially not reproducible - 2 users at two different points in time may end up with a different workspace for the same project.

For majority of users coming to che i'm sure they are more interested in having something that works with their project rather than guaranteed reproducible environments - they live with this just fine in all other IDE's. I know eclipse che wants to offer 100% reproducible environments and I think that is fine; but imposing that on every usecase I find very limiting. Especially newcomers and that we are forcing them to not only manually update their maven/gradle build setup mechanisms but then also have to remember to update the devfile.yml to match it....that should be possible to keep aligned or simply just not have to care.

Also "most probably prepared for you by someone else"
a) majority of users coming to Che will not have that and I can tell you the pre-defined we have now are so static and outofdate it prooves it is not scaling as it is now

I personally don't see it this way. I may be wrong here, but the use case of having a clickable button on github is a nice thing to have but it is not the main use case of Che, which I personally see more behind a corporate firewall - or rather with projects that strongly favor reproducibility and consistency in developer workspaces.

Again, not disagreeing here but I'll say that even in those environments having the metadata be part of the project is a very convenient and easy way to get started. like with other IDE's you have parts of your config in each project and then the IDE adapts to that rather than che's appraoch where a single file actually physically redefines the running IDE. it’s a very limiting setup when enforced at all times.

The pre-defined devfiles being out of date actually means that they are reproducible :) Yes, we could make a better job at keeping our stacks (i.e. ready to use initial project templates) up-to-date with the upstream but we need to do that in a versioned way.

sure - that shouldn't prevent me from saying something like "gradle:latest" in my devfile so i don't have constantly update it IMO.

b) I here are describing exactly the case where I 'm trying to setup a project or if I absolutely must a devfile that I can give to someone else for them to share...the format of the devfile prevents that by assuming the people I share with will always use my fork.

I think we've made at least some progress with #15765 which enables you to use a project-less devfile in your repo and deducing the project in the factory upon import (e.g. you point Che to your github repo and it will use it as the project if there are no other in the devfile).

great!

If a devfiles continues to be only for tons of hardcoded references then I believe che needs something else/new to become useful.

I think the reproducibility of the workspace environment is actually a feature, not a bug.

i didn't say it was a bug - I said if devfile insists on being hardcoded we'll need some other mechanism to allow easy import/use - I'm here assuming we want che to be usable as an online ide not just something for use behind a strictly controlled company firewalled environment.

It is absolutely wrong to enforce on users to have to figure out the detailed docker/tooling environment up front and IF you really wanted to that it is killing uptake that these configurations are not even shareable or in anyway possible to parameterise to it can be used inside and outside teams.

We could and should do a better job at helping people author the devfile and I think autodetection is a big part of that. We just took the easier path to having a featureful workspace description and that is "hardcoding" the workspace environment. On the other hand we do prepare a rather large set of predefined "stacks" which people can use to kickstart their development and fine-tune their devfiles based on their needs.

I can honestly say I completely ignore/avoid those as much as possible because they are so many and never seem to be something that matche sthe project(s) I want to import - either because out of date or because I have both a java app and a front end app....

So yes, if autodetection could be done instead and then configure a devfile on the fly which I could then optionally commit if I cared about 100% reproducability rather than "give me the latest greatest autodetect" that would be great.

I do agree with you that we should have a much better experience when initiating a project from existing sources without a devfile. Currently, we're trying to solve that by having the stacks that you base your initial project on and add your sources. Maybe we should start thinking about adding autodection to the picture. But as you very well know, autodetection is hard and never perfect.

no, and it does not need to perfect - just good enough and you can do that fairly easily at least for java and node.js world.

I can go and use Eclipse, vscode, netbeans, gitpod.io, vscode online and a bunch of other IDE's instantly with projects today and none of them needs this kind tooling specific setup to get started but ALL of them have ways to after the fact customize, store, share configurations etc.

In my experience with IDEs that is not true. I am clearly not able to install stock Eclipse and start to work on say a Node.js application without any additional setup.

I was thinking primarily java as that’s the majority ...but even node.js today is fairly good in eclipse with its auto-install of plugins as you start opening node/javascript related files it adapts.

Che requires rebuilds/restarts here...

Eclipse may provide syntax highlighting out of the box, but I am pretty sure it doesn't understand the >project setup without any additional plugins. I am also sure that my Eclipse installation is going to be >different from my colleagues'. But devfile doesn't deal with just the IDE which we've been concentrating >on here. I can also define the database server that my application needs, or in a microservice world - I can declare the cooperating microservices that I am not working on directly as just components running within the workspace and I don't need to deal with setting them up or building them at all. IMHO no IDE does that as of yet and no autodetection can do that either.

no, and all of that are great and awesome - but that level of complexity we should ensure does not leak into new or casual users experience.

and i'll just point out that worlds IDE's today works pretty great without having these features so lets be careful.

Che is the one that is standing out making it extremely hard to use because it insists on its configuration to be defined upfront and not be shareable. If the devfile is not way to do that then I suggest we find something else to use to allow easy and shareable confgurations.

We were bad at sharability using forks (and note that behind a corporate firewall, you don't use forks all that much in my humble experience) and based on your great input, I think we've improved the situation.

forks is just one aspect where it becomes very clear we can do better. the issues above exists no matter if you have a fork or not.