Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update operator-sdk to the latest version #19466

Closed
tolusha opened this issue Mar 31, 2021 · 1 comment
Closed

Update operator-sdk to the latest version #19466

tolusha opened this issue Mar 31, 2021 · 1 comment
Assignees
Labels
area/che-operator Issues and PRs related to Eclipse Che Kubernetes Operator kind/enhancement A feature request - must adhere to the feature request template. severity/P1 Has a major impact to usage or development of the system.
Milestone

Comments

@tolusha
Copy link
Contributor

tolusha commented Mar 31, 2021

Is your enhancement related to a problem? Please describe.

For the time being che-operator uses operator-skd v0.17.2.
It is quite old. The latest version is v1.5.0

Describe the solution you'd like

Update operator-sdk to the latest v1.5.0 version

It is prerequisite for #19408
Since DWCO [1] has a newer project structure we have to update operator-sdk and che-operator project structure to easily merge codebase.

[1] https://github.com/che-incubator/devworkspace-che-operator

@tolusha tolusha added kind/enhancement A feature request - must adhere to the feature request template. severity/P1 Has a major impact to usage or development of the system. area/che-operator Issues and PRs related to Eclipse Che Kubernetes Operator labels Mar 31, 2021
@tolusha tolusha added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Mar 31, 2021
@AndrienkoAleksandr AndrienkoAleksandr removed the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Mar 31, 2021
@l0rd l0rd added new&noteworthy For new and/or noteworthy issues that deserve a blog post, new docs, or emphasis in release notes and removed new&noteworthy For new and/or noteworthy issues that deserve a blog post, new docs, or emphasis in release notes labels Apr 1, 2021
@tolusha tolusha added this to the 7.30 milestone Apr 12, 2021
@tolusha tolusha mentioned this issue Apr 12, 2021
63 tasks
@AndrienkoAleksandr AndrienkoAleksandr self-assigned this Apr 26, 2021
@tolusha tolusha mentioned this issue Apr 29, 2021
41 tasks
@tolusha tolusha modified the milestones: 7.30, 7.31 May 5, 2021
@tolusha tolusha mentioned this issue May 24, 2021
47 tasks
@tolusha tolusha modified the milestones: 7.31, 7.32 May 24, 2021
@tolusha tolusha mentioned this issue Jun 11, 2021
3 tasks
@AndrienkoAleksandr
Copy link
Contributor

During work on this issue we faced with a very big dependency upgrade. We analyzed about 580 dependencies(initially it was even 616), created green list of them:

https://github.com/eclipse-che/che-operator/blob/6dcdf13b8ad69981c457cca69024c93d90544120/Dependencies.md

For another dependencies we created plenty big CQ list:

https://gist.github.com/AndrienkoAleksandr/eddc8b47a1510585e70386a2d4af4df3

We wrote and generated using script(https://github.com/AndrienkoAleksandr/clearly-defined-scripts/blob/main/createClearDefinedCurations.sh) 192 curation pull requests for 197 dependencies to make “clearlydefined” analyze of this dependencies... And we are awaiting to merge these pr's to see the full picture: https://github.com/clearlydefined/curated-data/pulls/AndrienkoAleksandr

And I wrote a draft script https://github.com/AndrienkoAleksandr/clearly-defined-scripts/blob/main/validateDependencies.sh . This script should validate golang dependencies for projects which support go modules. So we want to automate dependency checks, because manual checks aren't reliable and require impressive a lot of the time. This time would be better spent for development.

== Implementation details validateDependencies.sh
=== Introduction
This script should work with modern golang projects which use golang modules. Go modules dependency manager for now is standard for golang language and that's a bad idea to use something else. Each project which uses this standard should have go.mod and go.sum files. go.mod file dependencies list is defined by the developer manually or with help of IDE plugins. Very often developers define here only straight dependencies, without transitive dependencies. go.sum contains full list dependencies, even transitive and is auto-generated, so developers shouldn't edit this file manually.

Proposal (that's my own opinion)
I would recommend Eclipse developers to pin in the go.mod file all dependencies using the replace directive to freeze all of them(even transitive). Otherwise developers could get a minor upgrade of one of the dependencies or even few, and this minor dependency could grab new unexpected transitive dependencies, which developers didn't check using https://clearlydefined.io or Eclipse CQ. If you freeze dependencies you don't need to check the go.sum file often, because the state of the file will be stable(you will see changes only for new dependencies). About minor upgrade dependencies, if we want to have proposals with minor upgrade fixes from time to time, I think this stuff should be automated somehow using CI and that's another topic which shouldn't be mixed with stable dependency graph.... Also "freezed" go.mod file can be good if since a lot of the time you want to compile some old version and you want to do that without upgrade surprises(when compilation fails due bad minor upgrade, caused by mistake of transitive dependency developers)... Also it will be a super stable combination to use freezed go.mod file and vendor folder. Vendor folder should also help to hold transitive dependencies in long term scope without surprises(when somebody removed github repo with source code or moved it without redirect, or moved to the gitlab or something like that).

=== Implementation
Script generates golang modules dependencies file: https://github.com/AndrienkoAleksandr/clearly-defined-scripts/blob/main/validateDependencies.sh#L123 . This file that's a list of golang dependencies imports with versions. One dependency or package of the dependency per line.

Notice: instead of generating a dependency list file, theoretically, we can use a go.sum file. But it's hard to parse it using shell. We have to use code with programming language to achieve it. Or we have to find and use cli binary to use it in the script.

Then script using golang imports evaluates real source locations AndrienkoAleksandr/clearly-defined-scripts@2538658#diff-e65f4ea80c1f3af3549f7dc08b6b452681e71e4294d7b83aa854cdbdd6fce5ceR59 . After that script retrieves commit hash for dependency AndrienkoAleksandr/clearly-defined-scripts@2538658#diff-851887bef5577b975f44b02ae208ccb9ca3bb6614bbf14a86f6900f31997d764R162-R171 . Then we make request to “clearlydefined” api to estimate dependency licence rate:
AndrienkoAleksandr/clearly-defined-scripts@2538658#diff-e65f4ea80c1f3af3549f7dc08b6b452681e71e4294d7b83aa854cdbdd6fce5ceR167 . Clearlydefined.io doesn't have a separate dependency type for golang. That's why in the request we are using type "git" and provider "github.com". At the end of the June 2021 “cleardefined” developers will provide support for "gitlab" git provider too.

Notice: clearlydefined git provider supports also tags in semver format vx.x.x
but all golang dependencies which I saw used commit hash. Only for some dotnet dependencies I saw using tags. Maybe we should use both tags and hashes, but in my script implementation I used only hashes.
Notice(general info about valid release tag for golang dependency): if golang project has release tag without letter 'v' - that's a wrong version for go mod.file. go mod commands will fail. Developer should use hash instead of tag for such "bad" releases.

=== Migration this script to the dependency analyze tool

It would be nice to migrate this script to the java code implementation of the Eclipse project https://github.com:eclipse/dash-licenses.git. I think we can do that in two steps:

  1. We have to write GoSumFileReader.java parser to parse dependencies from go.sum file. And then reuse “clearlydefined” code to validate dependencies. If dependency isn't green we should throw a warning about that.
    Optional: Also the tool can find and propose nearest golang versions with good license rate and print them for the developer. Developer could try this version manually in the go.mod file. If code will be compilable and doesn't fail in runtime, then the developer could use this proposed dependency. I think that it's not easy to automate auto-compilation to propose only compilable versions, because the tool doesn't know how to compile each project. But maybe we can do it like some executable argument. And recompilation is time consuming. So, that's area for discussion.
  2. Take a look at what we have to do to automatically create a CQ checks on the Eclipse ipteam gitlab repository for golang dependencies (experimental feature).

One more optional proposal. Should be separated issue and topic for discussion:
We could write feature for dash-license project which will create automatically pull requests with “clearlydefined” currations for dependency versions absent on this resource https://github.com/clearlydefined/curated-data. Some draft I wrote here: https://github.com/AndrienkoAleksandr/clearly-defined-scripts/blob/main/createClearDefinedCurations.sh

Remark: From the box, for me is strange, why dash license project has licence rate is 60: https://github.com/eclipse/dash-licenses/blob/master/core/src/main/java/org/eclipse/dash/licenses/ISettings.java#L22 . From Eclipse docs it should be 75: https://www.eclipse.org/projects/handbook/#ip-clearlydefined-using. I know using env developer could change this value, but it would be nice to set up actual value from the box... Fix me if I'm wrong.

===Limitations:

  1. Such implementation should support all modert golang projects, because using go modules is standard since golang 1.11-1.12. But this implementation shouldn't support old projects without modules. Even theoretically such support could be hard, because there were a lot of golang dependencies managers: godep, glide, gopm and so on. I think we should force Eclipse developers to use new golang features instead of trying to support analyzing outdated archived dependencies managers.
  2. This implementation will support dependencies which stored source code on the github(in the nearest feature we can cover gitlab too). If developers have dependencies with any other location, then the tool should throw an unsupported warning for each such dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/che-operator Issues and PRs related to Eclipse Che Kubernetes Operator kind/enhancement A feature request - must adhere to the feature request template. severity/P1 Has a major impact to usage or development of the system.
Projects
None yet
Development

No branches or pull requests

3 participants