Fix JobDSL requiring empty and UI-only values #254

bitwiseman · 2019-11-05T01:13:58Z

JENKINS issue(s):
- https://issues.jenkins-ci.org/browse/JENKINS-59321
Description:
- The change to use repository URL as a textbox also change the DataboundConstructor. This change means that JobDSL users scripts were broken and they are force to enter blank fields that should be UI implementation details. This change fixes this by switching to a constructor with no arguments and then does different things for UI and JobDSL scenarios. For UI, it uses bindResolve to access the passed JSON and select which settings to use. For JobDSL, it lets the caller provide the information they choose.
Documentation changes:
- None. This is an implementation detail that returns the JobDSL behavior to what it was before the issue was introduced. NOTE: in that sense this is a breaking change from v2.5.9, but that is because this was a breaking change from v2.5.3.
Users/aliases to notify:
- @joseblas @car-roll @kshultzCB @olamy @dwnusbaum

Change to fix one thing your break several others.

bitwiseman · 2019-11-14T01:12:23Z

@darxriggs
Could I ask a favor of you? This PR would be good to get into the next release - it would make JobDSL folks happy. But I don't have the bandwidth to work on it. Would you have any time?

darxriggs · 2019-11-16T10:18:53Z

I can keep this PR in mind but cannot make any promises when I'll get around to have a look at it in more detail.

bitwiseman · 2020-01-14T22:22:43Z

Reporting user said this works as expected.
https://issues.jenkins-ci.org/browse/JENKINS-59321?focusedCommentId=383056&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-383056

car-roll

👍

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

darxriggs · 2020-01-30T21:18:20Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+    public GitHubSCMSource bindResolve(StaplerRequest staplerRequest, JSONObject json) {
+        boolean configuredByUrl = json.getBoolean("configuredByUrl");
+
+        if(configuredByUrl) {


There should be an extra space after if.

I so want to turn on automatic formatting on this project.

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

darxriggs · 2020-01-30T21:44:35Z

...va/org/jenkinsci/plugins/github_branch_source/GitHubBranchSourcesJCasCCompatibilityTest.java

@@ -37,6 +37,6 @@ protected void assertConfiguredAsExpected(RestartableJenkinsRule restartableJenk

    @Override
    protected String stringInLogExpected() {
-        return "Setting class org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.repoOwner = jenkins-infra";
+        return "Setting org.jenkinsci.plugins.github_branch_source.GitHubSCMSource{id='null'}.repoOwner = jenkins-infra";


Why is that change required / where is it originating from?

I think the toString() is generated from some underlying library. Do you need me to go digging?

toString() is from jenkins.scm.api.SCMSource. I am just wondering why this was different before. I guess there are changes (e.g. configuration-as-code upgrade) merged from master into this branch that are not easily visible in the GitHub UI.

darxriggs · 2020-01-31T22:27:26Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+     *
+     * @param repoOwner the repository owner.
+     * @param repository the repository name.
+     * @param repositoryUrl HTML URL for the repository. If specified, takes precedence over repoOwner and repository.


I guess HTML should be HTTP here.

darxriggs · 2020-01-31T22:42:44Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

@@ -413,10 +431,7 @@ public void setApiUri(@CheckForNull String apiUri) {
            return;


An exception could be thrown to make this invalid usage more clear.

No, we actually don't want to throw an exception here.
One the things that is most striking about this PR is how much of it goes against what we all agree is "good design" because it is serving the needs of data binding. This is another one of those cases. If this gets called more than once due to databinding we don't want to throw.

In what scenarios are the setters called more than once? That seems surprising to me, is it related to bindResolve? Otherwise, I would think that we definitely would want to throw exceptions as @darxriggs describes, otherwise, something like this ordering of events (only possible via incorrect data binding): setRepoOwner, followed by setRepositoryUrl, followed by setRepository, would silently leave the GitHubSCMSource in an invalid state. Is there a guaranteed order in which setters will be called?

darxriggs · 2020-01-31T22:52:18Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+
+            try {
+                info = GitHubRepositoryInfo.forRepositoryUrl(repositoryUrl);
+            } catch (Throwable e) {


Catching a Throwable is not a good idea. What's the intention here?

No matter what error you get from this method call, report it to the UI as a form validation error. It will always look better that way than the default UI exception formatting.

ah, and e.getCause(); returns a throwable.

darxriggs · 2020-01-31T22:56:17Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

@@ -1996,37 +2078,54 @@ public FormValidation doCheckCredentialsId(@CheckForNull @AncestorInPath Item co
        public FormValidation doValidateRepositoryUrlAndCredentials(@CheckForNull @AncestorInPath Item context,
                                                                    @QueryParameter String repositoryUrl,
                                                                    @QueryParameter String credentialsId) {
+
+            // Start by verifying we have a well-formed GitHub API URL


Was there a reason to move this check before the permission check? Even if verifying a URL should not denote a security risk.

darxriggs · 2020-01-31T23:10:44Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

                GHRepository repo = github.getRepository(info.getRepoOwner() + "/" + info.getRepository());
                if (repo != null) {
                    sb.append(" Connected to ");
                    sb.append(repo.getHtmlUrl());
                    sb.append(".");
                }
            } catch (IOException e) {
-                return FormValidation.error(e, "Error validating repository information. " + sb.toString());
+                return FormValidation.error(e, sb.toString() + " Error connecting to repository: " + e.getMessage());


Is the "successful" information in sb in an error case that useful in the UI?

The sb is built up along the way. It lets people understand how far verification got before failing.

darxriggs · 2020-02-01T09:04:08Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

@@ -371,14 +405,14 @@ public GitHubSCMSource(@CheckForNull String id, @CheckForNull String apiUri,
                           @NonNull String checkoutCredentialsId,
                           @CheckForNull String scanCredentialsId, @NonNull String repoOwner,
                           @NonNull String repository) {
-        this(repoOwner, repository, null, false);
+        this(repoOwner, repository);


The question here is if a with @Deprecated annotated constructor should be used (even if this is a very valid one) or just the default one + setters. Otherwise code checkers / IDEs might complain about this fact.

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

darxriggs · 2020-02-02T14:24:55Z

I manually tested using this branch and mvn hpi:run with the following JobDSL examples. These are the 3 ways I am aware of to create a GitHub multi-branch pipeline.

example-1 and example-2 is working fine. But example-3 has an issue. It creates a job without the required settings. Debugging this shows that setRepositoryUrl() is not called for whatever reason.

example-1

JobDSL

multibranchPipelineJob('example-1') {
  branchSources {
    github {
      id('1')
      repoOwner('jenkinsci')
      repository('job-dsl-plugin')
    }
  }
}

Job XML Output

...
<jenkins.branch.BranchSource>
    <source class="org.jenkinsci.plugins.github_branch_source.GitHubSCMSource">
        <id>1</id>
        <scanCredentialsId/>
        <checkoutCredentialsId>SAME</checkoutCredentialsId>
        <repoOwner>jenkinsci</repoOwner>
        <repository>job-dsl-plugin</repository>
        <includes>*</includes>
        <excludes/>
        <buildOriginBranch>true</buildOriginBranch>
        <buildOriginBranchWithPR>true</buildOriginBranchWithPR>
        <buildOriginPRMerge>false</buildOriginPRMerge>
        <buildOriginPRHead>false</buildOriginPRHead>
        <buildForkPRMerge>true</buildForkPRMerge>
        <buildForkPRHead>false</buildForkPRHead>
    </source>
    <strategy class="jenkins.branch.DefaultBranchPropertyStrategy">
        <properties class="empty-list"/>
    </strategy>
</jenkins.branch.BranchSource>
...

example-2

JobDSL

multibranchPipelineJob('example-2') {
  branchSources {
    branchSource {
      source {
        github {
          id('2')
          repoOwner('jenkinsci')
          repository('job-dsl-plugin')
        }
      }
    }
  }
}

Job XML Output

...
<jenkins.branch.BranchSource plugin="branch-api@2.1.2">
    <source plugin="github-branch-source@2.5.9-SNAPSHOT" class="org.jenkinsci.plugins.github_branch_source.GitHubSCMSource">
        <id>2</id>
        <apiUri>https://api.github.com</apiUri>
        <repoOwner>jenkinsci</repoOwner>
        <repository>job-dsl-plugin</repository>
        <traits/>
    </source>
</jenkins.branch.BranchSource>
...

example-3

JobDSL

multibranchPipelineJob('example-3') {
  branchSources {
    branchSource {
      source {
        github {
          id('3')
          repositoryUrl('https://github.com/jenkinsci/job-dsl-plugin.git')
        }
      }
    }
  }
}

Job XML Output

...
<jenkins.branch.BranchSource plugin="branch-api@2.1.2">
    <source plugin="github-branch-source@2.5.9-SNAPSHOT" class="org.jenkinsci.plugins.github_branch_source.GitHubSCMSource">
        <id>3</id>
        <apiUri>https://api.github.com</apiUri>
        <repoOwner/>
        <repository/>
        <traits/>
    </source>
</jenkins.branch.BranchSource>
...

darxriggs · 2020-02-02T19:23:48Z

I digged a bit deeper regarding the failing example-3.

With some debugging and changing the log level for org.jenkinsci.plugins.structs.describable I found the root cause.

The job is not created correctly via Job DSL because of the customizations in GitHubSCMSource.DescriptorImpl implementing the CustomDescribableModel which are removing repositoryUrl and configuredByUrl.

call chain

log

Feb 02, 2020 8:11:09 PM FINE org.jenkinsci.plugins.structs.describable.DescribableModel
class org.jenkinsci.plugins.github_branch_source.GitHubSCMSource$DescriptorImpl translated {traits=[], id=3, repositoryUrl=https://github.com/jenkinsci/job-dsl-plugin.git} to {id=3, traits=[]}

This functionality is completely new to me. Maybe @dwnusbaum and @joseblas can assist here as they have been involved in this customization.

darxriggs · 2020-02-02T21:27:11Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+     * @param repositoryUrl HTTP URL for the repository.
+     * @since 2.2.0
+     */
+    @Deprecated


Is it really required to be deprecated or just not intended to be used for data-binding? It could be used programatically without problems.

It could be used, but we do not want that. Also, as far as I know, there aren't any other valid uses for these constructors aside from data binding.

I think it would be ok to leave it undeprecated, even if just for use in tests to avoid misuse of the data binding constructor. Same with new GitHubSCMSource(String, String). If you leave it deprecated, you will get community PRs that convert all of the tests to use the non-deprecated constructor, which may or may not be what you want.

darxriggs · 2020-02-02T21:27:26Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

@@ -353,7 +362,32 @@ public GitHubSCMSource(String repoOwner, String repository, String repositoryUrl
     */
    @Deprecated


Is it really required to be deprecated or just not intended to be used for data-binding? It could be used programatically without problems.

bitwiseman · 2020-02-03T05:39:35Z

@darxriggs
Thanks for the in-depth testing. I will update to fix what you found.

dwnusbaum

Seems ok in general, I added some minor comments. I am a bit surprised by your comment that the setters can be called twice in some scenarios, so I am going to do a bit of testing.

dwnusbaum · 2020-02-03T17:04:22Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+     * @param repositoryUrl HTTP URL for the repository.
+     * @since 2.2.0
+     */
+    @Deprecated


I think it would be ok to leave it undeprecated, even if just for use in tests to avoid misuse of the data binding constructor. Same with new GitHubSCMSource(String, String). If you leave it deprecated, you will get community PRs that convert all of the tests to use the non-deprecated constructor, which may or may not be what you want.

dwnusbaum · 2020-02-03T17:05:32Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+        repoOwner = "";
+        repository = "";


Pretty weird, might mask some bugs. Personally, I would rather see an NPE caused by the instance being in an illegal state than have the instance silently do the wrong thing without any way for us to diagnose the issue.

Maybe assign the fields to null and add SuppressFBWarnings mentioning that the setters will make the fields non-null in practice, or change the annotation to @CheckForNull (but that's going to require a bunch of changes in the methods here). Same thing for the defaultIfBlank calls in the setters.

@dwnusbaum Community PR to do that conversion are fine with me. 😄

As to the assigning blank, you and @darxriggs both commented on it. And you're right, but ... this exactly the point of this PR - DataBinding with a complex set of parameters basically demands that we construct the instance in non-ready-to-run state. On the other hand, for any valid use of of an instance these values will not be null. On the other other hand, we don't make them null by default because we want to be able to load/save instances that are in a non-ready-to-run state so that we can modify/fix them to be ready-to-run.

dwnusbaum · 2020-02-03T17:10:21Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

@@ -413,10 +431,7 @@ public void setApiUri(@CheckForNull String apiUri) {
            return;


In what scenarios are the setters called more than once? That seems surprising to me, is it related to bindResolve? Otherwise, I would think that we definitely would want to throw exceptions as @darxriggs describes, otherwise, something like this ordering of events (only possible via incorrect data binding): setRepoOwner, followed by setRepositoryUrl, followed by setRepository, would silently leave the GitHubSCMSource in an invalid state. Is there a guaranteed order in which setters will be called?

dwnusbaum · 2020-02-03T17:15:08Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+            this.setRepositoryUrl(json.get("repositoryUrl").toString());
+        } else {
+            this.repositoryUrl = null;
+            this.apiUri = json.get("apiUri").toString();


Suggested change

this.apiUri = json.get("apiUri").toString();

this.apiUri = setApiUri(json.get("apiUri").toString());

Otherwise we never go through normalizeApiUri on this path.

dwnusbaum · 2020-02-03T17:19:54Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+                if (e instanceof IllegalArgumentException && e.getCause() != null) {
+                    e = e.getCause();
+                }


What's this for? If it's to try to make it easier for users to understand the stack trace, I would delete these lines, and change the next line from e.getMessage to something like The format of the repository URL is invalid. Please check the help text and adjust the URL, which is probably more helpful unless we are dealing with a bug.

Also though, if the changes in doValidateRepositoryUrlAndCredentials aren't required for the main fix, it might be better to move it to a separate PR.

Okay, you've both said this less than ideal. I see a way to clean it up.

This allows repositoryUrl to be passed in via JobDSL and templating, but filters it out on round trip.

bitwiseman · 2020-02-03T19:33:46Z

@dwnusbaum

Is there a guaranteed order in which setters will be called?

Short answer, no. If there is an order it is incidental.

bitwiseman · 2020-02-03T20:21:15Z

I was hoping to include this in today's release, but discussion indicates there's more work to be done here.
I'll circle back to this after releasing v2.6.0.

…bSCMSource.java

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

…bSCMSource.java

bitwiseman added 3 commits November 4, 2019 16:36

JENKINS-59321 Check Url well-formed

e9fd081

Fix constructor requiring too many arguments

d9af47c

Fixes for other compatibility issues

09ba8e4

Change to fix one thing your break several others.

bitwiseman requested a review from dwnusbaum January 24, 2020 22:00

bitwiseman marked this pull request as ready for review January 24, 2020 22:04

bitwiseman added 2 commits January 27, 2020 00:31

Added tests

1bdcb6c

Merge remote-tracking branch 'jenkinsci/master' into issue/JENKINS-59321

d3097a5

bitwiseman requested a review from joseblas January 27, 2020 09:28

car-roll approved these changes Jan 27, 2020

View reviewed changes

bitwiseman added 2 commits January 28, 2020 14:39

Merge remote-tracking branch 'jenkinsci/master' into issue/JENKINS-59321

b0c2bf5

Merge remote-tracking branch 'jenkinsci/master' into issue/JENKINS-59321

c69569b

darxriggs reviewed Jan 31, 2020

View reviewed changes

bitwiseman changed the title ~~Issue/jenkins 59321~~ Fix JobDSL requiring empty and UI-only values Feb 1, 2020

Changed based on code review feedback

c1ab4d8

darxriggs reviewed Feb 1, 2020

View reviewed changes

darxriggs reviewed Feb 2, 2020

View reviewed changes

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java Outdated Show resolved Hide resolved

darxriggs reviewed Feb 2, 2020

View reviewed changes

dwnusbaum reviewed Feb 3, 2020

View reviewed changes

Fix repositoryUrl handling

5474da9

This allows repositoryUrl to be passed in via JobDSL and templating, but filters it out on round trip.

bitwiseman force-pushed the issue/JENKINS-59321 branch from 4b6100d to 5474da9 Compare February 3, 2020 18:12

More changes for feedback

dead063

bitwiseman added 2 commits January 26, 2021 13:55

Merge branch 'master' into issue/JENKINS-59321

262e8fc

Update src/main/java/org/jenkinsci/plugins/github_branch_source/GitHu…

9155051

…bSCMSource.java

bitwiseman commented Jan 26, 2021

View reviewed changes

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java Outdated Show resolved Hide resolved

bitwiseman added 5 commits January 26, 2021 14:08

Update src/main/java/org/jenkinsci/plugins/github_branch_source/GitHu…

d8f52a3

…bSCMSource.java

Base commit for auto formatting for open PRs

801806c

Merge branch 'task/formatting-base' into issue/JENKINS-59321

203bf54

Apply autoformatting

df13c57

Merge remote-tracking branch 'upstream/master' into issue/JENKINS-59321

7bc0d08

bitwiseman closed this Nov 21, 2023

		@@ -413,10 +431,7 @@ public void setApiUri(@CheckForNull String apiUri) {
		return;

		@@ -353,7 +362,32 @@ public GitHubSCMSource(String repoOwner, String repository, String repositoryUrl
		*/
		@Deprecated

	this.apiUri = json.get("apiUri").toString();
	this.apiUri = setApiUri(json.get("apiUri").toString());

Fix JobDSL requiring empty and UI-only values #254

Fix JobDSL requiring empty and UI-only values #254

Conversation

bitwiseman commented Nov 5, 2019 • edited

bitwiseman commented Nov 14, 2019

darxriggs commented Nov 16, 2019

bitwiseman commented Jan 14, 2020

car-roll left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitwiseman Feb 1, 2020 • edited

Choose a reason for hiding this comment

darxriggs Feb 1, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitwiseman Feb 1, 2020 • edited

Choose a reason for hiding this comment

bitwiseman Feb 1, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darxriggs commented Feb 2, 2020 • edited

example-1

JobDSL

Job XML Output

example-2

JobDSL

Job XML Output

example-3

JobDSL

Job XML Output

darxriggs commented Feb 2, 2020

Choose a reason for hiding this comment

bitwiseman Feb 3, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitwiseman commented Feb 3, 2020

dwnusbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bitwiseman commented Feb 3, 2020

bitwiseman commented Feb 3, 2020

bitwiseman commented Nov 5, 2019 •

edited

bitwiseman Feb 1, 2020 •

edited

darxriggs Feb 1, 2020 •

edited

bitwiseman Feb 1, 2020 •

edited

bitwiseman Feb 1, 2020 •

edited

darxriggs commented Feb 2, 2020 •

edited

bitwiseman Feb 3, 2020 •

edited