Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: add a build tag into the version for non-release builds #156

Closed
isaac-io opened this issue Sep 14, 2022 · 7 comments · Fixed by #157
Closed

build: add a build tag into the version for non-release builds #156

isaac-io opened this issue Sep 14, 2022 · 7 comments · Fixed by #157
Assignees
Labels
build Build related enhancement New feature or request
Milestone

Comments

@isaac-io
Copy link
Contributor

For various tasks (especially for the performance tests) we sometimes need to compare builds and tell how they are ordered in the git tree. The current version number is not enough because it's only changed on releases, and while the git hash uniquely identifies a build, it requires access to the git tree in order to find that information, and it becomes useless if the specific commit is later amended or deleted from the tree altogether.

Add an optional build tag to the version and allow setting it during the build (or at configuration time, if using CMake). If it's not set, use a script to calculate a build tag based on the git tree since the last release, falling back to a build tag of ? in case we aren't able to determine the git tree, which would basically mean that this isn't a release build, and that the git hash should be manually checked as we do today.

@isaac-io isaac-io self-assigned this Sep 14, 2022
@isaac-io isaac-io added enhancement New feature or request build Build related labels Sep 14, 2022
isaac-io added a commit that referenced this issue Sep 14, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, it will be calculated using the
state of the git tree since the last release tag (for example, for this
PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the branch name can not be determined, a question mark will be
used instead.
isaac-io added a commit that referenced this issue Sep 14, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, it will be calculated using the
state of the git tree since the last release tag (for example, for this
PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
isaac-io added a commit that referenced this issue Sep 14, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
@isaac-io isaac-io linked a pull request Sep 14, 2022 that will close this issue
isaac-io added a commit that referenced this issue Sep 14, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
isaac-io added a commit that referenced this issue Sep 15, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
isaac-io added a commit that referenced this issue Sep 15, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
@isaac-io
Copy link
Contributor Author

@rtpro @VodOleg please review #157.

@isaac-io isaac-io added this to the v2.1.0 milestone Sep 21, 2022
isaac-io added a commit that referenced this issue Oct 20, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
@isaac-io isaac-io modified the milestones: v2.1.0, v2.2.0 Oct 26, 2022
@isaac-io isaac-io assigned erez-speedb and unassigned rtpro and VodOleg Oct 26, 2022
@erez-speedb
Copy link

This tag is now passable and in the collected metadata.

isaac-io added a commit that referenced this issue Nov 4, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
@mrambacher
Copy link
Contributor

Is there a better way of doing this? On my machine this adds 60+ seconds to the make command before we start even start the build. Is there other information we can collect or a means to collect it faster?

On branches that were committed, the "build date" should have been close to what was required -- it is supposed to capture the date/time of the last git commit.

Also, this PR modifies the "RocksDB version" string when I believe it means to modify the "Speedb" one...

@isaac-io
Copy link
Contributor Author

isaac-io commented Nov 8, 2022

Is the slowdown happening even in incremental builds, or only after a make clean? If the former, we can add caching of the calculated tag so that it's only recalculated if HEAD is different than what it was when the tag was calculated. If the latter, I don't have a good solution for that, unfortunately, because we do need to calculate the tag somehow. The idea is not just to capture the date/time, but to allow easily comparing two builds and quickly inferring where they are on the git tree in relation to each other.

As for the version, we actually only have a Speedb version string, since this function is only used for outputting to the LOG file and the tools help text, so we don't really care about backward compatibility, and the format is <speedb-version> (<rocksdb-version>) (currently 2.1.0 (7.2.2)).

@mrambacher
Copy link
Contributor

The slowdown happens every time.
time make -j 32 dbg
$DEBUG_LEVEL is 2
Makefile:178: Warning: Compiling in debug mode. Don't use the resulting binary in production

  • GEN make_config.mk
    make: Nothing to be done for `dbg'.
    make -j 32 dbg 75.01s user 28.32s system 92% cpu 1:51.49 total
    markrambacher@marks-mbp speedb % time make -j 32 dbg
    $DEBUG_LEVEL is 2
    Makefile:178: Warning: Compiling in debug mode. Don't use the resulting binary in production
  • GEN make_config.mk
    make: Nothing to be done for `dbg'.
    make -j 32 dbg 75.22s user 28.54s system 92% cpu 1:52.38 total

The tag that is generated is also kind of strange. It is meaningless to me:
#define SPDB_BUILD_TAG "(#merge-7.8+108)-(yuval/comp_precut2-squashed-10655+7)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+7)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+7)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+54)-(#merge-7.8+4)-(yuval/comp_precut2-squashed-10655+25)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+55)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+48)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+3)-(#merge-7.8+4)-(yuval/comp_precut2-squashed-10655+199)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+7)-(#merge-7.8+2)-(yuval/comp_precut2-squashed-10655+4)-(#merge-7.8+2)-(yuval/comp_precut2-squashed-10655+7)-(#merge-7.8+1)-(yuval/comp_precut2-squashed-10655+5)-(#merge-7.8+2)-(yuval/comp_precut2-squashed-10655+7)-(#merge-7.8+69)-(main+1)-(#merge-7.8+3)-(main+1)-(#merge-7.8+7)-(main+1)-(#merge-7.8+1)-(main+3)-(#merge-7.8+3)"
static const std::string speedb_build_tag = "speedb_build_tag:" SPDB_BUILD_TAG;

@isaac-io
Copy link
Contributor Author

isaac-io commented Nov 9, 2022

I see. Yeah, that's a bug in the script. I opened #225 to track it and submitted #226 to fix it (I also added caching of the build tag while at it in order to speed up the build in case the git history hasn't changed).

@isaac-io
Copy link
Contributor Author

Woops. I just realised that you also correctly spotted the issue with the version string. I assumed that you meant the GetRocksBuildInfoAsString(), and that's indeed the only function that exists. However, the changes were indeed incorrectly applied to GetRocksVersionAsString() instead of GetSpeedbVersionAsString(). My bad. I opened #231 to track it and submitted #233 to to fix it. Thanks for catching this!

Yuval-Ariel pushed a commit that referenced this issue Nov 15, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
Yuval-Ariel added a commit that referenced this issue Nov 23, 2022
The change set includes:
1) A header file that contains the Major, Minor, and Patch versions of speedb
as a macro, as well as a set of functions returning info about how/when/where
this version of speedb was created.

also includes all changes done on build_version.cc.in in the following commits:
1. version: remove superfluous build property
The `speedb_build_spdb_key` property is unused and was accidentally
imported as part of #1.

2.general: replace RocksDB references in strings with Speedb (#64)
This includes references in statuses as well as tools output.

3.build: add a version build-tag for non-release builds (#156)
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.
If the git tree state can not be determined, a question mark will be
used instead.
build_version: apply the build tag to the Speedb version string (#231)

4. The changes in #157 were accidentally applied to the `GetRocksVersionAsString()`
function instead of the `GetSpeedbVersionAsString()` function. This replaced
the RocksDB patch number with the Speedb one, and added the build tag in
the wrong place.
Fix it by moving the logic to the intended function.
Yuval-Ariel pushed a commit that referenced this issue Nov 25, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
Yuval-Ariel added a commit that referenced this issue Nov 25, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
Yuval-Ariel added a commit that referenced this issue Nov 25, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
Yuval-Ariel pushed a commit that referenced this issue Nov 25, 2022
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
Yuval-Ariel pushed a commit that referenced this issue Apr 30, 2023
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
Yuval-Ariel pushed a commit that referenced this issue May 4, 2023
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
udi-speedb pushed a commit that referenced this issue Nov 13, 2023
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
udi-speedb pushed a commit that referenced this issue Nov 15, 2023
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
udi-speedb pushed a commit that referenced this issue Dec 3, 2023
The build tag can be set during the build (using either the Makefile
or the CMake). If it's not provided, and we're not in a release build,
it will be calculated using the state of the git tree since the last
release tag (for example, for this PR the build tag will be calculated as
`(main+17)-(156-build-add-a-build-tag-into-the-version-for-non-release-builds+1)`.

If the git tree state can not be determined, a question mark will be
used instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Build related enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants