-
Notifications
You must be signed in to change notification settings - Fork 632
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] Enable Dependabot on the repository #4720
Comments
We have many PRs submitted by dependabot, but we can't merge it directly, because it will cause dependency&license check fail. Do you have any suggestions? |
We have a merge-dependabot-reusable.yaml Github Actions script that merges Dependabot PR's automatically if they pass the tests. @vy, any ideas how to adapt this to EventMesh? The main problem, as I see it that EventMesh has a binary distribution with third-party dependencies, while Log4j doesn't. Therefore your merge Dependabot script must be more complex to adhere to the Apache "Assembling LICENSE and NOTICE' policy. I can try to enhance your IIRC Apache Airflow also has a complex CI deployment to upgrade several Python packages a day. @potiuk, is there some part of your infrastructure that can be adapted to a Java project? |
If I understand it right, for a particular dependency (e.g., Log4j), these four locations must be aligned:
[Please correct me if I am missing any above.] You can follow a strategy as follows:
If you are interested in, I can implement this as a contractor and contribute to EventMesh. Disclaimer: I am a long time Log4j contributor and Apache Logging Services (log4j, log4cxx, etc.) PMC Chair. |
another location shall be aligned: Thank you for proposing the solution. I find it feasible. @xwm1992 @qqeasonchen , do you think we can further streamline the five files mentioned above? |
For licensing problems there is also an Apache Creadur Whisker project, although I don't know what is it worth. |
In your experience and knowledge, do you think our LICENSE and dependency management are excessively configured? For example, we attach a txt file for every third-party dependency (e.g., Regarding dependency management, I haven't seen any other projects using the |
@Pil0tXia, I wouldn't say so. There are no plug-n-play tooling to make the life easier for 1) Gradle/Maven application developers 2) that need to provide a binary distribution 3) complying with the ASF distribution requirements. Airflow, Kafka, etc. they all roll out their own solutions. These efforts are mostly manual, that is, people occasionally check the validity of these files and update them as they see fit. Note that point (2) – that is, the fact that you distribute binaries – is crucial. For instance, Log4j doesn't distribute binaries containing dependencies. Hence, there, we don't have this problem. We only maintain a
No, I don't think so.
I don't think so. You have license check as a part of the build. This is great. You also partially implemented a framework to automatically include the LICENSE, etc. files while creating a distribution. But updates to this framework are not automated yet. In conclusion, I don't think you are over-engineering. I think you are trying to do the right thing. I will consult to |
For libraries that don't bundle their dependencies there is some automatic tooling: For example the last one in
and we don't have to do anything to generate it (the I believe that for projects that bundle their dependencies a similar mechanism could be used. Notice files are more complex (but fortunately uncommon), but licenses IMHO could be listed in a file like the example above (and bundled in a single copy). IANAL |
I have shared this issue in |
@Pil0tXia, I would like to share some updates from the
I figured that license information is available in I am still interested in this gig. Let me know what you think. |
I agree with your idea. Thank you very much for the insights you've provided. When EventMesh creates a distribution, we execute the |
May we make some progress here? |
@Pil0tXia, who is expected to deliver the work? As I have stated earlier, I can implement this feature as a contractor – i.e., paid job. Yet I haven't been approached in this direction. |
@vy |
I think dependabot does not support editing |
No, it doesn't support your My approach to this issue would be to:
This should be a rather safe setup since:
Remark: the main disadvantage of this setup is that you manage through Dependabot all your dependencies, both direct and transitive. You have about 300 deps, which means around 10-20 Dependabot PRs daily! The automatic merge is really required. |
An alternative approach would be through SBOM generation:
|
I found that the plugin https://github.com/jk1/Gradle-License-Report has been introduced in the project to generate license reports. However, license files are exported to the While the html report it generates declares 300+ dependencies in full, it does not provide the contents of the LICENSE file for each one. Trying to generate a LICENSE file for a project using html as a data source is difficult. The existing LICENSE file for the project has only 200+ dependencies in it, which is also incomplete. It seems that this plugin is hardly a help for generating LICENSE files. However, the json file generated by cyclonedx-gradle-plugin is too huge. I am writing gradle scripts based on its data format. #4827 mainly addresses the dependencies problem. I'll address license problem in another PR. |
Hi, I've made several attempts at apache/eventmesh-dashboard#122, apache/eventmesh-dashboard#128 and apache/eventmesh-dashboard#130. It looks like the most we can do is auto-approve a Dependabot PR, but can't do merge it. I looked at merge-dependabot-reusable.yaml. It deals with git, bypassing GitHub's PR mechanism in favor of committing directly on the master branch. I'm not sure if the |
The The automation of Dependabot merges is not so trivial: it took @vy a week or two of full-time work to figure out all the details (fortunately we had a sponsor for that). I was than able to reproduce it in half a day in copernik-eu/log4j-tomcat/.github/workflows/merge-dependabot.yaml. The basic idea is that we follow the manual procedure to merge PRs:
There are some pitfalls you should be aware of:
Debugging tipsIt is usually easier to test automatic Dependabot merges in your personal fork, since you can add your GPG key as repository secret. If you run out of Dependabot PRs, you can:
|
* Sync changes in #4719 * minor change * Only keep the artifact name * Run `sed -i 's/-[0-9].*\.jar//g'` * Run `sort known-dependencies.txt | uniq > known-dependencies-unique.txt` * Allow CI to run on branches with namespace in the branch name in forked repos * Correct typo and remove useless command * Use `sort -u -o` instead of `uniq` to remove duplicate artifacts with different version * Enlarge open-pull-requests-limit * minor: polish tips * Test apache/skywalking-eyes/dependency CI result * Fix 'unable to find version `0.6.0`' * See debug log to prove it works * skywalking-eyes/dependency doesn't support gradle, test basic actions/dependency-review-action * Add all denied licenses * Remove redundant check * Remove not included SPDX: ASL, RSAL * Add a useful printAllDependencyTrees task * Exampt safe artifact under multiple licenses * Exempt more safe artifacts (Looks like the last of them) * 'allow-dependencies-licenses' attribute only supports single-line text * Add a TODO comment * Add more file extensions for checkstyle * Resolve some checkstyle header violations * Add back apache/skywalking-eyes * Fix downloaded file didn't have a `.` * Disable Go deps update & Must pass CI before merge * No need to force up-to-date & Auto-approve only * Remove the slash at the end of the homepage url in Repo GitHub desc * Skip patch updates temporarily to reduce PR noise * Logback removed after be06ef7 * Accept patch update * Submit dependency graph * Follow https://github.com/gradle/actions/blob/main/docs/dependency-submission.md#usage-with-pull-requests-from-public-forked-repositories * try to sort dependency graph workflow exec seq * `workflow_run` event will only trigger a workflow run if the workflow file is on the default branch * Grant required permission of CodeQL * Attempt to fix 'No dependency graph files found to submit' * Attempt to fix 'No dependency graph files found to submit' try 2 * Attempt to fix 'No dependency graph files found to submit' try 3 * Attempt to fix 'No dependency graph files found to submit' try 4 * Try to check dependency-review * Only check bundled dependencies * Fix 'No snapshots were found for the head SHA' attempt 1 * Test runtimeClasspath dependencies * Revert "Test runtimeClasspath dependencies" This reverts commit 3de89a5. * Try to retry 1 hr wo wait for snapshot update * Test gradle/actions#196 (comment) * Add todo comments * Keep implementation and compileOnly for now * Keep runtimeOnly deps * [Breaking Change] Remove dependency-review-action and wait for its bugfix * Add checkDeniedLicense into CI * minor code optimization
Search before asking
Environment
Other
EventMesh version
Other
What happened
Since EventMesh has a lot of dependencies and the upgrades are done manually, the TAR distribution contains many libraries with known vulnerabilities.
For examples Jackson 2.13.0 is included multiple times in the distribution.
One way to solve these problems is to enable Dependabot (cf. documentation) or another dependency manager on the repository. Dependabot is highly configurable. You can upgrade every dependency or just those that are vulnerable.
How to reproduce
Check the contents of the distribution archive and look for Jackson 2.13.0.
Debug logs
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: