Unexpected merge importing a pull request

[ob]

When trying to import a pull request using the workflow:

core.workflow(
    name = "import_pr",
    mode = "CHANGE_REQUEST",
    origin = git.github_pr_origin(
        url = github_public_url,
        branch = "master",
    ),
    destination = git.destination(
        url = github_sot_url,
        fetch = "master",
    ),
    origin_files = glob(["**"]),
    destination_files = glob(
        ["somedir/**"],
        exclude = [
            "somedir/bfile",
        ],
    ),
    authoring = authoring.pass_thru("Default email <default@defatult.com>"),
    transformations = [
        core.move("", "somedir"),
        metadata.save_author(),
        metadata.expose_label("COPYBARA_INTEGRATE_REVIEW"),
        metadata.expose_label("GITHUB_PR_NUMBER", new_name = "Closes", separator = " #"),
    ],
)

I get:

% copybara copy.bara.sky import_pr 1
Nov 09, 2019 2:51:27 PM com.google.copybara.Main configureLog
INFO: Setting up LogManager
Copybara source mover (Version: Unknown version)
Task: Integrating change from https://github.com/ob/cb-sot-public/pull/1 from ob:ob-patch-1 21171d13a57703bae681c45dd57b2e76bffb1ab5
ERROR: Conflict detected while rebasing /Users/obonilla/copybara/temp/workdir109556842962639907/checkout to d0b3be83ebc842c8dd6feae051391d7a82adf9b8. Please sync or update the change in the origin and retry. Git output was:
First, rewinding head to replay your work on top of it...
Applying: Project import generated by Copybara.
Applying: Project import generated by Copybara.
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
CONFLICT (add/add): Merge conflict in .gitignore
Auto-merging .gitignore
Patch failed at 0002 Project import generated by Copybara.
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

However, that pull request didn't change the .gitignore file so I didn't expect this to be a problem.

SoT repo: https://github.com/ob/cb-sot, exported repo: https://github.com/ob/cb-sot-public, PR that I'm trying to import: https://github.com/ob/cb-sot-public/pull/1

I don't see why Copybara is trying to merge the two .gitignore files, are they treated specially by globs?

[ob]

Looking at the git history for the generated project, I see

% git log --oneline --graph
*   af64a61 (HEAD) Merge pull request #1 from ob:ob-patch-1
|\
| * 21171d1 (refs/copybara_fetch/refs/pull/1/head) Update afile
| * 55cc1e4 Project import generated by Copybara.
| * 80b7e76 Project import generated by Copybara.
* d0b3be8 (copybara/resolvedPush-dacca1cc-cf6b-45dc-976c-189a5df3f5aa, refs/copybara_fetch/master) add somedir/.gitignore
* b4e91bd add some files
* 36eb4a9 Initial commit

Looking at the two tips of the commit history:

% git ls-tree -r d0b3be8
100644 blob 82f3a88e17b409ae206b19e55f5c5b23eb83bcc1	.gitignore
100644 blob 261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64	LICENSE
100644 blob 0f19791fc0d465f4226ee44b07a50dbad869e493	README.md
100644 blob 86e041dad66a19b9518b83b78865015f62662f75	somedir/.gitignore
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	somedir/afile
100644 blob 257cc5642cb1a054f08cc83f2d943e56fd3ebe99	somedir/bfile

which is the state of my SoT, and

% git ls-tree -r 21171d1
100644 blob 86e041dad66a19b9518b83b78865015f62662f75	.gitignore
100644 blob 3bd1f0e29744a1f32b08d5650e62e2e62afb177c	afile

and the merge:

% git ls-tree -r af64a61
100644 blob 86e041dad66a19b9518b83b78865015f62662f75	.gitignore
100644 blob 3bd1f0e29744a1f32b08d5650e62e2e62afb177c	afile
100644 blob 86e041dad66a19b9518b83b78865015f62662f75	somedir/.gitignore
100644 blob 3bd1f0e29744a1f32b08d5650e62e2e62afb177c	somedir/afile

It's almost like the core.move() operation here is not removing the source files.

[ob]

One thing to note when running this again with -v and reading the logs is this:

Executing [git '--git-dir=/Users/obonilla/copybara/cache/git_repos/https%3A%2F%2Fgithub%2Ecom%2Fob%2Fcb-sot%2Egit' '--work-tree=/Users/obonilla/copybara/temp/workdir606592250108702709/checkout' merge-base d0b3be83ebc842c8dd6feae051391d7a82adf9b8 21171d13a57703bae681c45dd57b2e76bffb1ab5]
Command 'git' finished in 00:00.011. Process exited with status 1

Not sure if that's related though...

[mikelalcon]

Hi Oscar, Sorry for the late reply. The reason is a conflict with the labels because this is git to git (we haven't used this much, we almost always use it for non-git to/from git). The way to fix it is for your CHANGE_REQUEST workflow use: destination = git.destination( url = "https://github.com/ob/cb-sot", fetch = "master", integrates = [], ), The default for integrates is to do a merge with the change from COPYBARA_INTEGRATE_REVIEW. But in this case it is the change that we are importing itself. Note that integrates = [], should only be set in the CHANGE_REQUEST one. The one from cb-sot -> cb-sot-public should rely on the default (and have the same expose label).

…

On Thu, Nov 14, 2019 at 5:29 PM Oscar Bonilla ***@***.***> wrote: One thing to note when running this again with -v and reading the logs is this: Executing [git '--git-dir=/Users/obonilla/copybara/cache/git_repos/https%3A%2F%2Fgithub%2Ecom%2Fob%2Fcb-sot%2Egit' '--work-tree=/Users/obonilla/copybara/temp/workdir606592250108702709/checkout' merge-base d0b3be83ebc842c8dd6feae051391d7a82adf9b8 21171d13a57703bae681c45dd57b2e76bffb1ab5] Command 'git' finished in 00:00.011. Process exited with status 1 Not sure if that's related though... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#104?email_source=notifications&email_token=AAHJWQCPEE2WYM2K7276EYTQTXGK3A5CNFSM4JLJ6XHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEDQ6FI#issuecomment-554110741>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHJWQDUUBDOYPCKZOV2RJ3QTXGK3ANCNFSM4JLJ6XHA> .

[ob]

No worries Mikel… Maybe I’m not understanding the flow then. I tried adding the `integrates` attribute, and as you said, it fixed the merge problem but I’m not getting the behavior I expect. What I’m trying to do is set up a workflow where I have an internal private git repo and an external public git repo. The private one has more than just the piece I’m moving to the public one, thus the `core.move()` calls. The flow I want is an initial export of the history with some cleanups. Then the public one should become the SoT and the internal one would be more of a testing ground for public PRs. Does that make sense? I’m having trouble understanding all the labels and attributes in the workflows and what they mean :-) Thanks for your help!

…

On Nov 14, 2019, at 8:02 PM, Mikel Alcon ***@***.***> wrote: Hi Oscar, Sorry for the late reply. The reason is a conflict with the labels because this is git to git (we haven't used this much, we almost always use it for non-git to/from git). The way to fix it is for your CHANGE_REQUEST workflow use: destination = git.destination( url = "https://github.com/ob/cb-sot", fetch = "master", integrates = [], ), The default for integrates is to do a merge with the change from COPYBARA_INTEGRATE_REVIEW. But in this case it is the change that we are importing itself. Note that integrates = [], should only be set in the CHANGE_REQUEST one. The one from cb-sot -> cb-sot-public should rely on the default (and have the same expose label). On Thu, Nov 14, 2019 at 5:29 PM Oscar Bonilla ***@***.***> wrote: > One thing to note when running this again with -v and reading the logs is > this: > > Executing [git '--git-dir=/Users/obonilla/copybara/cache/git_repos/https%3A%2F%2Fgithub%2Ecom%2Fob%2Fcb-sot%2Egit' '--work-tree=/Users/obonilla/copybara/temp/workdir606592250108702709/checkout' merge-base d0b3be83ebc842c8dd6feae051391d7a82adf9b8 21171d13a57703bae681c45dd57b2e76bffb1ab5] > Command 'git' finished in 00:00.011. Process exited with status 1 > > Not sure if that's related though... > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#104?email_source=notifications&email_token=AAHJWQCPEE2WYM2K7276EYTQTXGK3A5CNFSM4JLJ6XHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEDQ6FI#issuecomment-554110741>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAHJWQDUUBDOYPCKZOV2RJ3QTXGK3ANCNFSM4JLJ6XHA> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[mikelalcon]

I see. Forget about the integrates then. For some reason I though your SoT was the private repo. Then you wouldn't need to use a CHANGE_REQUEST workflow (For now). This is what I would setup: 1. One workflow from private -> public in ITERATIVE mode. You probably want to use --ignore-noop. Throw away this workflow once the initial import is done. 2. Once you have public ready, make public SoT (For public part of the repo) 3. Create an ITERATIVE or SQUASH workflow from public -> private. git.origin -> git.destination. This will be in charge of importing submitted code internally. At this point you can work in the external repo and periodically import to the internal repo. Now presubmits: *Checking that external changes don't break internal:* 4. Configure an external -> internal workflow with mode CHANGE_REQUEST_FROM_SOT. This will be git.github_pr_origin to git.github_pr_destination. Don't merge this prs, use them for testing only (we can do something about this later). *Checking that internal changes don't break external and do changes to external from internal:* 5. Configure an internal to external workflow using CHANGE_REQUEST mode. use git.github_pr_origin to git.github_pr_destination. You don't submit in the the internal repo. You merge/submit in the destination (External). For 5) sometimes you need to do an incompatible change that needs to change internal an external. You can chose between breaking the internal repo temporarily or (What we usually do) when doing an import from step 3), manually add the pending incompatible commits from internal. I hope it helps! On Fri, Nov 15, 2019 at 12:41 PM Oscar Bonilla <notifications@github.com> wrote:

…

No worries Mikel… Maybe I’m not understanding the flow then. I tried adding the `integrates` attribute, and as you said, it fixed the merge problem but I’m not getting the behavior I expect. What I’m trying to do is set up a workflow where I have an internal private git repo and an external public git repo. The private one has more than just the piece I’m moving to the public one, thus the `core.move()` calls. The flow I want is an initial export of the history with some cleanups. Then the public one should become the SoT and the internal one would be more of a testing ground for public PRs. Does that make sense? I’m having trouble understanding all the labels and attributes in the workflows and what they mean :-) Thanks for your help! > On Nov 14, 2019, at 8:02 PM, Mikel Alcon ***@***.***> wrote: > > Hi Oscar, > > Sorry for the late reply. > > The reason is a conflict with the labels because this is git to git (we > haven't used this much, we almost always use it for non-git to/from git). > > The way to fix it is for your CHANGE_REQUEST workflow use: > > destination = git.destination( > url = "https://github.com/ob/cb-sot", > fetch = "master", > integrates = [], > ), > > The default for integrates is to do a merge with the change > from COPYBARA_INTEGRATE_REVIEW. But in this case it is the change that we > are importing itself. > > Note that integrates = [], should only be set in the CHANGE_REQUEST > one. The one from cb-sot -> cb-sot-public should rely on the default (and > have the same expose label). > > On Thu, Nov 14, 2019 at 5:29 PM Oscar Bonilla ***@***.***> > wrote: > > > One thing to note when running this again with -v and reading the logs is > > this: > > > > Executing [git '--git-dir=/Users/obonilla/copybara/cache/git_repos/https%3A%2F%2Fgithub%2Ecom%2Fob%2Fcb-sot%2Egit' '--work-tree=/Users/obonilla/copybara/temp/workdir606592250108702709/checkout' merge-base d0b3be83ebc842c8dd6feae051391d7a82adf9b8 21171d13a57703bae681c45dd57b2e76bffb1ab5] > > Command 'git' finished in 00:00.011. Process exited with status 1 > > > > Not sure if that's related though... > > > > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub > > < #104?email_source=notifications&email_token=AAHJWQCPEE2WYM2K7276EYTQTXGK3A5CNFSM4JLJ6XHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEDQ6FI#issuecomment-554110741 >, > > or unsubscribe > > < https://github.com/notifications/unsubscribe-auth/AAHJWQDUUBDOYPCKZOV2RJ3QTXGK3ANCNFSM4JLJ6XHA > > > . > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub, or unsubscribe. > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#104?email_source=notifications&email_token=AAHJWQB44YEN7A7WZULKWGDQT3NLRA5CNFSM4JLJ6XHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEGFTZA#issuecomment-554457572>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHJWQDAK6722BMUGK64PO3QT3NLRANCNFSM4JLJ6XHA> .

[mikelalcon]

Hi Oscar, did it work for you?