Core submodule

[rhyolight]

Fixes #596 and #587.

This removes the nta directory entirely and replaces it with a git submodule with the same path for nupic.core. The build script has been updated to sync the submodule with each build.

[oxtopus]

This needs to be the unauthenticated URL:

https://github.com/numenta/nupic.git

Or:

git://github.com/numenta/nupic.git

[rhyolight]

Thanks.

[oxtopus]

My bad. Needs to be git://github.com/numenta/nupic.core.git

[oxtopus]

Builds for me, but fails in Travis for the reason I mentioned in another comment.

[rhyolight]

Some more cleanup to build process incoming.

[rhyolight]

Getting git error in Travis. Not sure why, because d8e805ff5265130da12980a44dcb535e60da4167 is definitely a tree in nupic.core. It's the tip of master.

$ git submodule update
Cloning into 'nta'...
remote: Reusing existing pack: 46611, done.
remote: Counting objects: 170, done.
remote: Compressing objects: 100% (170/170), done.
remote: Total 46781 (delta 142), reused 0 (delta 0)
Receiving objects: 100% (46781/46781), 195.35 MiB | 25.76 MiB/s, done.
Resolving deltas: 100% (27806/27806), done.
fatal: reference is not a tree: d8e805ff5265130da12980a44dcb535e60da4167
Unable to checkout 'd8e805ff5265130da12980a44dcb535e60da4167' in submodule path 'nta'
The command "git submodule update" failed and exited with 1 during checkout.

I'm going to attempt to replicate the Travis commands being executed locally to see if I can get the same error in a fresh checkout just like Travis does it.

[oxtopus]

Maybe disable Travis's automatic git submodule support, since we need to explicitly do it in build.sh

From http://docs.travis-ci.com/user/build-configuration/#Git-Submodules

git:
  submodules: false

[rhyolight]

@oxtopus Nice find! Trying now. I had the same error when running locally, however. Let's see what happens now.

[rhyolight]

Build seems to be making progress now.

[rhyolight]

Ready to merge.

[oxtopus]

Looks good to me.

How does this affect workflow if you need to work on both nupic and nupic.core? If a developer needs to make changes to nupic.core, and test it in nupic, how does one go about doing that?

[rhyolight]

I don't recommend editing a submodule in place within a parent module
(although I've done it successfully before, it can get confusing). The best
way to do this in my opinion is to checkout out both repos separately, then
within nupic edit the .gitsubmodule file to change the path for nupic.core
to point to the local checkout on your file system (you can use a relative
path). Then edit both projects, but push the changes to nupic.core first
before updating and committing the submodule SHA within nupic to point to
the new nupic.core.

---

Matt Taylor
OS Community Flag-Bearer
Numenta

On Mon, Feb 3, 2014 at 9:32 PM, Austin Marshall notifications@github.comwrote:

Looks good to me.

How does this affect workflow if you need to work on both nupic and
nupic.core? If a developer needs to make changes to nupic.core, and test it
in nupic, how does one go about doing that?

Reply to this email directly or view it on GitHubhttps://github.com//pull/597#issuecomment-34032106
.

[oxtopus]

I'd really like to know @scottpurdy's and especially @subutai's opinions on this as they are closer to this area of the code.

[rhyolight]

Sure. If we decide that the submodule workflow is too complex, we can remove it and replace with something we build that does the same thing.

[rhyolight]

I got into "submodule hell" at some point in the past, which put me off submodules for a long time, but here was the situation in that case. I had 3 projects, 2 of which depended on the same project.

parent-proj 
    ---> sibling-1
             ---> sibling-2
    ---> sibling-2

It was very common that I needed to change sibling-2 because of a requirement within parent-proj and/or sibling-1. This caused problems because every time I did this, I needed to commit and push to sibling-2, update the submodule reference in sibling-1 to point to the new SHA in sibling-2, then go to parent-proj and update the submodule reference for sibling-2 there as well and rebuild. It was a much more complex situation, and I ended up automating all the submodule commands necessary into a shell script that ran from the parent-proj.

This case, with a 1v1 dependency, is much simpler.

By default, submodules are in a detached HEAD state, so editing them in place can't happen until they are associated with a branch. In my workflow above, I did associate the submodule that way and work within the submodule inline within the parent project. So it is possible, but you have to remember to commit the changes from within the submodule up into the submodule's repository, then go back into the parent repository and commit/push the submodule change.

Another possibility is to edit the .gitmodules file within the parent project to reference a local checkout of the child module (but don't push the change to .gitmodules!). Then you can work on the child project direction from the checked out source code while making sure it doesn't break the parent project. Once you have what you want, push changes within the child project and update the submodule within the parent to point to the new SHA.

[subutai]

I don't really know much about submodules. I'm happy with any temporary stop-gap solution as long as it is short lived and doesn't affect the Grok pipeline. We can try out submodules and see how bad it is. I'm much more concerned with the end directory layout and where we end up, such as issue #591.

[rhyolight]

Ok, let's merge this then, and deal with the consequences. I want to start
hashing out nupic.core to build itself properly and nail down the
directory structures.

---

Matt Taylor
OS Community Flag-Bearer
Numenta

On Tue, Feb 4, 2014 at 9:58 AM, Subutai Ahmad notifications@github.comwrote:

I don't really know much about submodules. I'm happy with any temporary
stop-gap solution as long as it is short lived and doesn't affect the Grok
pipeline. We can try out submodules and see how bad it is. I'm much more
concerned with the end directory layout and where we end up, such as issue
#591 #591.

Reply to this email directly or view it on GitHubhttps://github.com//pull/597#issuecomment-34087689
.

[breznak]

hey :) sorry, I didn't follow up lately, just updated and was surprised by all nta/ removed. The build with submodule seems to work fine. Just needed to rm -rf the content of previous nta/, maybe an email or a wiki on "best practice" would be good.

[rhyolight]

I didn't have to rm -rf nta/* when I updated, but I'll send an email to the mailing list. This should only affect those of us who've already got a checkout.