Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set file encoding in GitLab commits #7381

Merged

Conversation

mikaellanger
Copy link
Contributor

request_{creator,updater}/gitlab.rb doesn't pass the DependencyFile content_encoding to the GitLab API. This causes vendored files (in our case yarn pnp zip files) to be uploaded as text.

@mikaellanger mikaellanger requested a review from a team as a code owner May 31, 2023 18:34
Copy link
Member

@jurre jurre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall changes look good, I made a few suggestions and had one question, lmk your thoughts.

Internally we have a few stability improvements that we need to focus on first before we can accept any contributions, so it'll take a bit of time before we can merge this, just FYI.

@@ -124,6 +125,15 @@ def file_action(file)
end
end

# @param [DependencyFile] file
def file_encoding(file)
if file.content_encoding == Dependabot::DependencyFile::ContentEncoding::BASE64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if file.content_encoding == Dependabot::DependencyFile::ContentEncoding::BASE64
if file.binary?

common/lib/dependabot/pull_request_updater/gitlab.rb Outdated Show resolved Hide resolved
@@ -108,7 +108,8 @@ def file_actions
{
action: file_action(file),
file_path: file.type == "symlink" ? file.symlink_target : file.path,
content: file.content
content: file.content,
**file_encoding(file)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we pass file.content_encoding directly? It would be either utf-8 or base64

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it like this for two reasons:

  1. I didn't want to change the behavior for the other case
  2. I don't want to make any assumption that the string values of constants defined for another purpose match the input required by the GitLab API, even if they happen to do so.

But I'll change it to pass file.content_encoding directly if you think that is better

@mikaellanger mikaellanger force-pushed the langer-patch-gitlab-content-type branch from 0d2b0db to 04a45f0 Compare June 15, 2023 16:28
@jurre jurre force-pushed the langer-patch-gitlab-content-type branch from 04a45f0 to 6804ba9 Compare June 20, 2023 13:32
@jurre jurre force-pushed the langer-patch-gitlab-content-type branch from 6804ba9 to 0deedaf Compare July 20, 2023 12:26
jeffwidman
jeffwidman previously approved these changes Jul 24, 2023
@jeffwidman jeffwidman dismissed their stale review July 24, 2023 06:54

Overlooked that Jurre's suggestions on binary weren't fully implemented... please do that first.

@mikaellanger
Copy link
Contributor Author

Overlooked that Jurre's suggestions on binary weren't fully implemented... please do that first.

@jeffwidman Not sure I understand what you mean? That code was removed in favor of setting encoding directly to file.content_encoding per the other comment by Jurre #7381 (comment)

@jeffwidman jeffwidman force-pushed the langer-patch-gitlab-content-type branch from a0173bf to e158497 Compare August 3, 2023 21:29
Copy link
Member

@jeffwidman jeffwidman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@jeffwidman jeffwidman enabled auto-merge (squash) August 3, 2023 21:30
@jeffwidman jeffwidman merged commit c991766 into dependabot:main Aug 3, 2023
107 checks passed
@fadedDexofan
Copy link

fadedDexofan commented Aug 18, 2023

@mikaellanger @jeffwidman GitLab API docs says that encoding should be either text or base64, not utf-8. Text is default. Setting encoding to utf-8 breaks dependabot.

/builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/gitlab-4.19.0/lib/gitlab/request.rb:71:in `validate': Server responded with code 400, message: actions[0][encoding] does not have a valid value. Request URI: <REPO_PATH>/repository/commits (Gitlab::Error::BadRequest)
from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/gitlab-4.19.0/lib/gitlab/request.rb:55:in `block (2 levels) in <class:Request>'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/gitlab-4.19.0/lib/gitlab/client/commits.rb:195:in `create_commit'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.226.0/lib/dependabot/clients/gitlab_with_retries.rb:67:in `public_send'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.226.0/lib/dependabot/clients/gitlab_with_retries.rb:67:in `block in method_missing'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.226.0/lib/dependabot/clients/gitlab_with_retries.rb:82:in `retry_connection_failures'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.226.0/lib/dependabot/clients/gitlab_with_retries.rb:64:in `method_missing'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.226.0/lib/dependabot/pull_request_creator/gitlab.rb:98:in `create_commit'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.226.0/lib/dependabot/pull_request_creator/gitlab.rb:42:in `create'
	from /builds/services-devops/kira-dependabot-script/vendor/ruby/3.1.0/gems/dependabot-common-0.226.0/lib/dependabot/pull_request_creator.rb:110:in `create'

brettfo pushed a commit to brettfo/dependabot-core that referenced this pull request Oct 11, 2023
Previously `request_{creator,updater}/gitlab.rb` didn't pass the DependencyFile `content_encoding` to the GitLab API. This caused vendored files (in our case `yarn` `pnp` zip files) to be uploaded as text.

---------

Co-authored-by: Mikael Langer <mikael.langer@safeture.com>
Co-authored-by: Jeff Widman <jeff@jeffwidman.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants