[WIP] Clone runtime and docs into build/runtime

[leonardehrenfried]

This implements waffle.io task #565.

I thought about using submodules but since the entire build folder is generated/copied I don't think it is a brilliant idea.

Let me know if something still needs work.

Thanks for your great work.

[mhinz]

The runtime target should do these 2 things:

1) add submodules (or some other form if more fitting) for vimscript and docs to build/runtime and build/runtime/docs respectively.
2) update submodules if they already exist

As you can see, Travis shows a failed build. Click on the Details button and find out why. :) In this case, mv fails because of an already existing directory.

[leonardehrenfried]

Are you sure that you want to add them as submodules?

In that case I think it would be better to clone them somewhere outside build and then copy/symlink them there.

[mhinz]

It's just that I used submodules when I prepared a patch for this issue, but use whatever seems most appropriate.

And I have no idea about Windows. Do Win filesystems support symbolic links?

[leonardehrenfried]

I think NTFS does but FAT doesn't. Or something like that. It's probably better to copy.

What would you think if I add the submodule to the top level folder and then copy them into build?

[mhinz]

Hmm, personally I would put them into build/. cmake -C build won't overwrite them, that's why subsequent calls to make runtime should update instead of removing build/runtime and readding it afterwards.

You probably will have to alter .gitignore then, too.

[leonardehrenfried]

I put everything under build and I squashed my commits. Please review.

[mhinz]

I think build/docs and build/vimscript can be removed afterwards, eh?

Otherwise it looks good to me.

[leonardehrenfried]

Either way is fine by me but didn't you want them to be updated if they
already exist?

On Sunday, April 27, 2014, Marco Hinz notifications@github.com wrote:

I think build/docs/ and build/vimscript can be removed afterwards, eh?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/633#issuecomment-41509511
.

[mhinz]

Via re-running make runtime, yup. But this way is cool with me too.

LGTM 👍

[leonardehrenfried]

I don't think I quite understand. We should have two targets? One for fetching and one for updating?

[tarruda]

LGTM, @jszakmeister what do you think about this?

[jszakmeister]

I guess two things come to mind:

1. Do we really want to embed the runtime and docs as submodules in this way? I'm a bit confused by the text in the PR:

I thought about using submodules but since the entire build folder is generated/copied I don't think it is a brilliant idea.

It appears to be a submodule to me, but this hints at it intending to be something else.

2. I don't like that the submodules live in the build area. They could live alongside other parts of the repo and we could copy the relevant portion of the trees into build.

3. I think the steps for this should probably live in CMakeLists.txt. It's not as easy as Make--I know--but the Makefile is really meant to be a thin veneer over CMake... just to make it easier for people to get up and running, not as a substitute for CMake.

It might be a good idea to consider using a shallow clone: git clone --depth 1 <URL>. I guess one question to ask, is do we want pin the neovim development against the runtime and docs? IIRC, submodules don't have the concept of a floating revision (e.g., follow master on neovim-docs, vs. checkout rev 12ab34cd of neovim-docs). So we'd have to periodically update the submodules to point to newer commits--which I think would not only get tiresome, but considering how fast the runtime is being updated, we might have many commits that are just "Updating runtime and docs."

So, the short form is that I'm not so sure about the approach (sorry @Lenniboy! I appreciate the contribution though!). @tarruda Do you have a vision about how things will get wrapped together, and how the runtime is going to be versioned with respect to the main neovim?

[leonardehrenfried]

Thanks, for the comments, @jszakmeister. I agree that the initial text is a little confusing. I was against using submodules, too, since they are quite complicated and fiddly to use and have the issues you mentioned (lots of manual updates necessary).

@mhinz thought that they are the way forward, though. If there is one person you need to convince him.

If we don't care about history we could also download the newest tarballs generated by Github rather than doing a shallow clone. (We would have to see what is faster.)

Anyway, I'm happy to implement whatever way is chosen in Cmake.

[mhinz]

I didn't say submodules are the way to go, but I used them when I prepared a patch before you took over. ;-) I said whatever seems the most approriate.

[leonardehrenfried]

I didn't mean to put the blame on you. Group hug!

I will just wait until there is a consensus on how it should happen.

[mhinz]

Didn't feel blamed, just wanted to point it out. ;>

I guess @jszakmeister has to decide this, since, as our official build master, he knows most about the build system and probably also has some future changes in mind.

[mhinz]

ping

[jszakmeister]

I guess @jszakmeister has to decide this, since, as our official build master, he knows most about the build system and probably also has some future changes in mind.

I don't think it's up to me alone. There are some upsides and downsides either way. Whichever way is chosen though, we need to have a way to build without the presence of Git. That way we can make release tarballs with the correct contents and not require anyone building from a tarball to download more content.

If we go with having submodules, I'm in favor of option (2): keeping it out of the build area and copying it in via CMake. Otherwise, I think we could add a target that will download and extract a couple tarballs from GitHub, but only if certain directories don't exists (perhaps a top-level runtime and docs folder). In which case, the top-level ones will be used instead.

@justinmk any thoughts on the matter?

[justinmk]

Why use submodules at all if it's just for building? What about git archive?

Answering my own question: if developer already has submodule pulled, then the next build only needs to pull the diff.

we need to have a way to build without the presence of Git

Will that include libuv source? Unless I missed discussion about this, how about a cmake target that uses git archive to pull libuv, neovim docs, neovim runtime, and creates a .tar.gz that we can post as a release?

If we go with having submodules, I'm in favor of option (2): keeping it out of the build area and copying it in via CMake

Yes. Whatever we use (git submodules, git archive, git subtree, ...?), my instinct would be to pull the components into the locations in the source tree where they would be if they were not separate repositories. Legacy Vim has this tree:

vim/
  src/
  runtime/
    doc/

So, I suggest pulling docs into neovim/runtime/doc, and vimscript into neovim/runtime/*.

neovim/
  src/
  runtime/ <-- neovim/vimscript repo
    doc/   <-- neovim/docs repo

If I'm missing anything, please let me know.

[leonardehrenfried]

I very much agree with @justinmk and I think @jszakmeister does too. So lets figure out which way we want them to be placed there.

I'm definitely not in love with git submodules and I've heard better things about subtrees, although it seems also not trivial to handle them correctly. I'm not aware of how git archive could help us pull dependencies. I thought it can only produce tarballs from your git tree.

Personally, I would vote for submodules. Any other opinions?

[jszakmeister]

we need to have a way to build without the presence of Git

Will that include libuv source? Unless I missed discussion about this, how about a cmake target that uses git archive to pull libuv, neovim docs, neovim runtime, and creates a .tar.gz that we can post as a release?

I consider libuv a different sort of dependency, like curses, so I had no intention of including the source in the final tarball. We currently have code to pull libuv via the deps mechanism, and it builds static libraries in the right format, if they want to go that route--though it may not be flexible enough for everyone. But, the runtime and docs are a different story, IMHO. They should be bundled into the final release tarball.

I'm not completely against submodules, but they are pegged against specific revisions. Which means they'll need to get updated often at the rate things are happening with Neovim. I've not tried nested submodules either, so I'm a bit concerned how well layering one inside the other will work, but maybe someone can play with that arrangement and give it a whirl?

Yes. Whatever we use (git submodules, git archive, git subtree, ...?), my instinct would be to pull the components into the locations in the source tree where they would be if they were not separate repositories.

/me agrees. BTW, do we know why they are separate repos at all?

[aktau]

/me agrees. BTW, do we know why they are separate repos at all?

@tarruda decided it to reduce management overhead (different people could manage different repositories).

[justinmk]

@aktau If GitHub's administration quirks were the only reason to separate the repositories, then I would campaign to re-integrate the repositories. We shouldn't be fiddling with git submodules just to get around GitHub's limitations. But I can think of another reason: we will have possibly many GUI frontends, which, presumably, would not live in the core repository, so even if we integrate the docs and vimscript repos back into core, we'll still have to deal with separate GUI (and who-knows-what-else) repos.

We haven't discussed what our release model will be. I assume (I don't know) that separate repositories may be helpful depending on our release model.

Created #700 to start discussion on the release model.

[justinmk]

Submodules or subtree seem to be the options. What we need now is a workflow, so that updating the docs, etc, is not a pain in the ass for devs who want to submit a patch. If a developer commits to core and needs to update the docs also, what would that look like at the command line (using submodules, using subtree)? I'm going to play with this (unless someone already has it nailed down...), and hope that exposes an obvious choice.

[eminence]

i'm not sure if my comment is totally relevant for this issue, but i'll bring it up anyway:

One problem with the current organization is that the separate repos to not encourage developers to update documentation when a feature is added/removed. Moving the doc (and to some extent runtime) repos into the main neovim repo would make it easier for PRs to include both code and docs. I'm not sure if there are other ways to solve this issue.

[tarruda]

One problem with the current organization is that the separate repos to not encourage developers to update documentation when a feature is added/removed. Moving the doc (and to some extent runtime) repos into the main neovim repo would make it easier for PRs to include both code and docs. I'm not sure if there are other ways to solve this issue.

Agreed. I added the docs/runtime to separate repositories to make better use of github permissions and delegate maintenance of documentation, but having to send multiple PRs is really inconvenient.

@justinmk What do you think about this? Should we merge the other repos into this one?

[justinmk]

@tarruda It's really tempting to do so (merge them into one), but give me until Monday to see if I can figure out an acceptable workflow using git submodules (I already determined that git subtree won't work for us). There could be other advantages to having separate repos, such as for licensing...

[tarruda]

@tarruda It's really tempting to do so (merge them into one), but give me until Monday to see if I can figure out an acceptable workflow using git submodules (I already determined that git subtree won't work for us). There could be other advantages to having separate repos, such as for licensing...

👍

[ghost]

Speculation:

And I have no idea about Windows. Do Win filesystems support symbolic links?

Unwarranted Assertion:

I think NTFS does but FAT doesn't. Or something like that. It's probably better to copy.

Yes, but nobody uses FAT for Windows anyway!

Initially NTFS had Junction, which was only comparable to symlink. Then they introduced Symbolic links in Vista onwards. See this @shanselman's post from last decade. Extract:

Postulate: Symlink is to Junction in Windows as Symlink is to Hardlink in Unix.

In comparison, VS2013 doesn't run on OS older than Windows 7. Meaning its quite safe to use symlinks in your case.

Secondly, if you want a real quick/swift way to: copy only changed stuff and just DO IT!, use robocopy; it will mirror the directories and subdirectories pronto (comparable to Unix rsync utility).

[justinmk]

@tarruda I'm finding it difficult to justify the separate repos.

• I don't see much advantage with respect to licensing as long as we keep a strict subdirectory separation in the main repo.
  • If we want to offer non-Apache plugins, etc., as separate downloads, the plugin manager can pull subdirectories from the core repo with git archive (or other methods), or we can publish the subdirectories to read-only repos.
• I'm not worried about administration because committers have shown good taste, restraint, and cooperation. Everyone understands the PR workflow (at least 2 LGTMs) and this prevents chaos from a bunch of unexpected commits.
• The separate repos aren't being maintained, probably because it isn't clear to the maintainers how to coordinate with the core.

git submodule workflow is problematic. From the git-scm book:

One common problem happens when a developer makes a change locally in a submodule but doesn’t push it to a public server. Then, they commit a pointer to that non-public state and push up the superproject. ... If that happens, you see an error ... Then, you e-mail that guy and yell at him.

The primary use case for submodules is to manage commits on a project you don't control. So adding this extra overhead for our group is hard to justify.

[felipecrv]

The primary use case for submodules is to manage commits on a project you
don't control. So adding this extra overhead for our group is hard to
justify.

Besides that, having docs and runtime in the same repo will make it easier
for PR reviewers to require the necessary changes in documentation right
away. The impression I have now is that we are accumulating pending
documentation changes. I didn't update the section about missing
'shortname' support and don't know if anyone did it.
On Jul 1, 2014 1:49 AM, "Justin M. Keyes" notifications@github.com wrote:

@tarruda https://github.com/tarruda I'm finding it difficult to justify
the separate repos.

• I don't see much advantage with respect to licensing as long as we
  keep a strict subdirectory separation in the main repo.
  • If we want to offer non-Apache plugins, etc., as separate
    downloads, the plugin manager can pull subdirectories from the core repo
    with git archive (or other methods), or we can publish the
    subdirectories to read-only repos.
    • I'm not worried about administration because committers have shown
      good taste, restraint, and cooperation. Everyone understands the PR
      workflow (at least 2 LGTMs) and this prevents chaos from a bunch of
      unexpected commits.
• The separate repos aren't being maintained, probably because it
  isn't clear to the maintainers how to coordinate with the core.

git submodule workflow is problematic. From the git-scm book
http://git-scm.com/book/en/Git-Tools-Submodules:

One common problem happens when a developer makes a change locally in a
submodule but doesn’t push it to a public server. Then, they commit a
pointer to that non-public state and push up the superproject. ... If that
happens, you see an error ... Then, you e-mail that guy and yell at him.

The primary use case for submodules is to manage commits on a project you
don't control. So adding this extra overhead for our group is hard to
justify.

—
Reply to this email directly or view it on GitHub
#633 (comment).

[justinmk]

closed by #938