Add library versioning and mark version 1.0.0-rc3

[lukeyeager]

Close #486, Close #2969, Close #3015, Close #3409

Originally NVIDIA#39 and NVIDIA#46

Included

Adds versioning to:

• shared library (libcaffe.so)
  • both build systems
• caffe binary (caffe --version)
• python module (caffe.__version__)

TODO

• matlab versioning (help?)

For discussion

• I'm assuming you want to be able to break backwards compatibility with MINOR releases (which means not following semantic versioning to the letter). If you want to go all the way with semver, we should set the SONAME to libcaffe.MAJOR instead of libcaffe.MAJOR.MINOR.
• Instead of this approach, you could follow the CMake stubs for how to do it with a header file - https://github.com/BVLC/caffe/blob/5a302a2b3b/cmake/Summary.cmake#L79-L84. I went this route due to some extra requirements for NVcaffe.

[lukeyeager]

/cc @CDLuminate for concerns w.r.t. debian packaging

/cc @Nerei for thoughts about the header file approach from #1667

[CDLuminate]

Well, that seems fine to me, and I'd be glad to import the officially versioned Caffe into the Debian package, with a terse version string like -1.0.0 rather than a snapshot version string like -0.9999~rc2+git2015MMDD-gABCDEFA. After importing it, likely that I just need to bump some numbers.

Thank you for working on versioning it :-)

By the way, Caffe itself currently doesn't block packaging at all (it was completed). The real trouble and the real blocker is CUDA from Debian Unstable/Experimental, which is quite out of date (6.5.14 currently) so that it refuses to work with GCC-5. And that is what I'm looking into recently.

[shelhamer]

@ronghanghu given your matcaffe contributions do you know how to mark the version in MATLAB?

[ronghanghu]

Hi, @shelhamer, I think for matlab we could simply do caffe.version similar to python. This should be quite easy.

[shelhamer]

Thanks @lukeyeager! This + the simple matcaffe version extension by @ronghanghu looks like all the infrastructure needed to declare a version.

[ronghanghu]

I think we can merge this PR first, and I will create another PR for matcaffe.

[lukeyeager]

Rebased after [my] conflicting change from #3127.

Any comments on the "for discussion" points above? I agree the matlab versioning can come later.

[flx42]

I think it's great, but for the future I would like to raise awareness on the possibility of creating some kind of config file for caffe (for instance caffe/config.h). This file would store all the relevant configuration variables that were used to build caffe.
When you use the C++ API of Caffe and include caffe.hpp, there is no way of knowing how the code was compiled. So you might end up using the same header but with a different environment that the one used when the library was compiled. This can break pretty badly if your code and the library disagree on the layout of the objects. For instance we have an example of this bug here:
NVIDIA#30

This file could look like this:

#define CAFFE_VERSION_MAJOR 1
#define CAFFE_VERSION_MINOR 0
#define CAFFE_USE_OPENCV 0
#define CAFFE_USE_CUDNN 1

[jczaja]

Hi,

I have a question regarding proposed changes. In CMakeLists.txt you are setting CAFFE_TARGET_VERSION and CAFFE_TARGET_SOVERSION. And then add_definitions is passing CAFFE_TARGET_VERSION.So it is like there is a versioning for caffe binary and separate one for caffe lib. This is fine, but in Makefile I can see only DYNAMIC_VERSION_{MAJOR|MINOR|REVISION} and then we have:
COMMON_FLAGS += -DCAFFE_VERSION=$(DYNAMIC_VERSION_MAJOR).$(DYNAMIC_VERSION_MINOR).$(DYNAMIC_VERSION_REVISION)

which is corressponding to setting vesion of caffe binary to same version as library is set to. Which seems like binary of caffe and library got the same version eg. its versioning is consistent. Please explain what Do I miss?

[Nerei]

@lukeyeager Delay with my thoughts was due to my vacations. Sorry.

For me hardcoding caffe version in headers is preferable than in build scripts. Most libraries use such way (at least from my own experience).

• it's single place to increment (now in make and cmake files). Headers are easily parsable (see aforementioned Summary.cmake#L79-L84) to propagate version info to build scripts.
• client libraries and apps that link/use caffe may implement different behavior and compatibility depending on caffe version definition. Binary distributions without build scripts are possible (say for cluster installs).

@flx42 I find such header useful, and it is generated already and stored in binary folder. See
CMakeLists.txt#L56. But might need to elaborate more, depending on what you want to do with it. Now it is considered as private.

[flx42]

Yes, I know it's easy to generate this header with autotools/cmake. But it will be more painful with the handwritten Makefile, unfortunately.

I think the use case I described should be a good motivating example, if you don't have a configuration file, using the C++ API can be dangerous when you have ifdefs in the header. It's also easier to check the version.

[lukeyeager]

@jczaja

With both build systems, the version reported with the CLI tool or with pycaffe should be MAJOR.MINOR.PATCH, and the SONAME should be MAJOR.MINOR. I believe I'm doing that correctly, though I admit it's less legible in the Makefile.

Are you seeing a bug, or are you just having trouble reading though the code? Either is potentially helpful feedback.

[lukeyeager]

@Nerei thanks for the response!

I agree it's relatively easy to parse the header file with CMake. I'll let the caffe devs weigh in on which approach they'd prefer.

[jczaja]

@lukeyeager

I observed potential issue , so I wanted to let you know about that. So I understand that regardless the build is created using Makefile or CMakeLists.txt , the result in terms of versioning should be the same. Let's consider example that build version is 1.2.0 , and caffe lib version is 1.0.0. Now :
A) CMakeLists.txt scenario:
set (CAFFE_TARGET_VERSION "1.2.0")
set (CAFFE_TARGET_SOVERSION "1.0.0")
add_definitions(-DCAFFE_VERSION=${CAFFE_TARGET_VERSION}) # Here we pass 1.2.0

B) MakeFile
DYNAMIC_VERSION_MAJOR := 1
DYNAMIC_VERSION_MINOR := 0
DYNAMIC_VERSION_REVISION := 0
...
COMMON_FLAGS += -DCAFFE_VERSION=$(DYNAMIC_VERSION_MAJOR).$(DYNAMIC_VERSION_MINOR).$(DYNAMIC_VERSION_REVISION) # Here we pass 1.0.0

So as You can see CAFFE_VERSION when using Makefiles would set to 1.0.0 , and when using CMakeLists.txt CAFFE_VERSION would be set to 1.2.0

Unless I misunderstood something in Makefiles, there is a discrepancy here. Please correct me If necessery

[cbalint13]

Dear All,

My +1 from release engineering point of view:

• That raw exposed Makefile, in a more proper manner, should be expanded normally from some Makefile.in using old style automake/auotoconf saga (old ways of building on unices). CMake for now do far better job, i believe it will overtake (or should) that Makefile. I also believe nobody will update Makefile to be expanded in proper ways using automake/autoconf tools (its painfull) as it would be standard.
• Any distro (linux) requires proper so-name versioning on libs as basic rule. Further packaging into .deb, .rpm and so on requires soversion at must level. Once proper so-versioning would be done, will open the path to packagers / distros.
• As stated, I believe that Makefile should die from a point, nobody will update it. CMake should overtake as primary build configurator / manager. By the way CMake in contrast with autoconf/automake and Make saga enable us to be cross-platform too.

Apologies if my +1 cent intervention looks odd, hope to be in community's service.

With respect,
~Cristian.

[lukeyeager]

@jczaja I'm assuming the CAFFE_TARGET_VERSION will always be MAJOR.MINOR.REVISION and the CAFFE_TARGET_SOVERSION will always be MAJOR.MINOR. The VERSION should never be 1.0.0 while the SOVERSION is 1.2.0 because those are different MINOR versions.

This would be equivalent:

# ---[ Caffe version
set(CAFFE_VERSION_MAJOR "1")
set(CAFFE_VERSION_MINOR "0")
set(CAFFE_VERSION_REVISION "0")
set(CAFFE_TARGET_VERSION "${CAFFE_VERSION_MAJOR}.${CAFFE_VERSION_MINOR}.${CAFFE_VERSION_REVISION}")
set(CAFFE_TARGET_SOVERSION "${CAFFE_VERSION_MAJOR}.${CAFFE_VERSION_MINOR}")
add_definitions(-DCAFFE_VERSION=${CAFFE_TARGET_VERSION})

[lukeyeager]

@cbalint13 I agree. I've created a new issue at #3351 since it's not necessarily related to this PR.

[lukeyeager]

Instead of waiting for 1.0 it seems like it could do good to adopt the new release workflow now for a series of release candidate (RC) versions while we cross off our 1.0 TODOs.
@shelhamer #3409 (comment)

That's fine with me. What would the tag be? I'd suggest v1.0.0-rc.3 since you already have an RC2.

[CDLuminate]

Any update to this ?

Currently I'm trying to help debian to update cuda to 7.5, and my local test shows that cuda 7.5 works
well with GCC-5. That means the time we upload the Caffe package gets one step closer.

So do you have any plan on next release of Caffe ?

If the next release date is not far from now, I'd like to wait for the new release and then upload the new release.

Thanks in advance.

[elezar]

Any chance that this could be merged. Would be nice to have version information for #3518.

[shelhamer]

Could this export __version__ directly instead of renaming in __init__.py?

[lukeyeager]

Yeah that'll work.

[shelhamer]

Drop in favor of exporting as __version__ on the boost::python side unless there's an obstacle there.

[longjon]

No need to duplicate these macros... why don't we put them in a common header file (e.g., common.hpp)?

[lukeyeager]

Can do.

[longjon]

The names are obscure here... how about calling the first macro STRINGIFY (which is the terminology from the cpp manual) and the second AS_STRING (which makes its functionality more clear)?

[lukeyeager]

Good idea.

[lukeyeager]

I've updated this PR with your latest suggestions. Any thoughts on the version (1.0.0 vs. 1.0.0-rc.3) or the SONAME (libcaffe.so.1.0 vs. libcaffe.so.1)?

[shelhamer]

@lukeyeager 1.0.0-rc3 makes sense to me so that we can adopt a versioning mechanism and take the next step from rc2. The SONAME could contain the full major/minor/patch version as that seems to be a common choice.

Although this will establish a version number, we have to come up with a policy for changing it. Releases can set the major and minor versions but the patch version needs to be automated given the rate of change of Caffe development. I'd like to highlight git describe and in particular git describe --tags for this purpose.

For the current HEAD this gives rc2-753-gcff6f3d where

• rc2 is the latest tag
• 753 is the number of commits since the tag
• gcff6f3d is the latest hash (prefixed with "g")

While the tagged release is a good shorthand version, it seems to me that the git describe version is the identifier that should be included with requests for help and the like.

As a follow-up the build should parse the number of commits from git describe to set the patch version.

[lukeyeager]

1.0.0-rc.3 makes sense to me
The SONAME could contain the full major/minor/patch version

Done.

As a follow-up the build should parse the number of commits from git describe to set the patch version.

git describe is a great solution. The CMake build is already using it when printing the project summary, so that should be pretty simple to implement.

[longjon]

This is included by caffe.hpp... I'd probably omit it here (unless lint really wants it).

[lukeyeager]

Oh yeah, duh.

[longjon]

As above.

[shelhamer]

@lukeyeager Just to check: should it be rc3 or rc.3? I expected rc3 and at the moment we have rc1 and rc2 but if the dot is common usage or there is another reason we could switch.

Thanks for the quick turnaround!

[lukeyeager]

Oh wait, sorry I misread your previous comment. You said you want MAJOR.MINOR.PATCH for the SONAME, not MAJOR or MAJOR.MINOR?

Gimme a minute, I'll have to rethink the symlinks ...

[lukeyeager]

Just to check: should it be rc3 or rc.3? I expected rc3 and at the moment we have rc1 and rc2 but if the dot is common usage or there is another reason we could switch.

I was going with Semantic Versioning conventions (http://semver.org/), but we seem to be throwing that out the window. I can do whatever you like.

[shelhamer]

I was going with Semantic Versioning conventions (http://semver.org/), but we seem to be throwing that out the window. I can do whatever you like.

Let's go with rc3 since we're not following Semantic Versioning. It's a nice specification but we don't need a further step in the pull request workflow. I like the plan of pinning PATCH to the number of commits since a release for now.

you want MAJOR.MINOR.PATCH for the SONAME, not MAJOR or MAJOR.MINOR?

MAJOR.MINOR.PATCH does look like the mode for library versions so let's go with that.

[lukeyeager]

Alright, I've updated it so that there's only one symlink and the SONAME is MAJOR.MINOR.PATCH:

libcaffe.so -> libcaffe.so.1.0.0-rc3

[shelhamer]

Thanks for the versioning @lukeyeager!

p.s. #486 needs a version header for close.

[Nerei]

shelhamer wrote:

Thanks for the versioning @lukeyeager!
p.s. #486 needs a version header for close.

@shelhamer @lukeyeager How about auto generation of caffe_version.h from cmake?
I can implement it by parsing "1.0.0-rc3".

[seanbell]

Perhaps the commit should be tagged, to make it easier to find releases (e.g. #3595)?