Serializing bitseq alloc #1788

abhi · 2017-05-31T20:42:01Z

Previously the bitseq alloc was allocating the first available bit from the beginning of the bit sequence for each sequence handle. With this commit the bitseq alloc will proceed
from the current allocation. This change will affect the way ipam and vni
allocation is done currently. The ip allocation will prcoeeed sequentially from the previous allocation as opposed to the first available IP. So even if the IPs are released it will not be reused until the allocator rolls over to the beginner of the bit sequence.

The commit also contains a fix for byteoffset allocation which plays a role in calculating the allocated ordinal. This scenerio play when the starting ordinal is part of the head sequence block and the available bit is part of subsequent blocks in the bitseq.

Signed-off-by: Abhinandan Prativadi abhi@docker.com

fcrisciani · 2017-06-05T16:21:04Z

bitseq/sequence.go

@@ -511,6 +513,29 @@ func getFirstAvailable(head *sequence, start uint64) (uint64, uint64, error) {
 	return invalidPos, invalidPos, ErrNoBitAvailable
 }

+//getAvailableFromCurrent will look for available ordinal from the current ordinal.


space missing

will make change

fcrisciani · 2017-06-05T16:21:10Z

bitseq/sequence.go

@@ -511,6 +513,29 @@ func getFirstAvailable(head *sequence, start uint64) (uint64, uint64, error) {
 	return invalidPos, invalidPos, ErrNoBitAvailable
 }

+//getAvailableFromCurrent will look for available ordinal from the current ordinal.
+// If none found then it will loop back to the start to check of the available bit.
+//This can be further optimized to check from start till curr in case of a rollover


space missing

fcrisciani · 2017-06-05T16:27:23Z

bitseq/sequence.go

-				if end < ret {
-					err = ErrNoBitAvailable
+				if err == nil {
+					h.curr = ret + 1


when the last bit is allocated is there the risk that doing the +1 the number wraps to 0?

that will result in failure and cause the allocation from the start=

fcrisciani · 2017-06-05T16:29:44Z

bitseq/sequence.go

+//This can be further optimized to check from start till curr in case of a rollover
+func getAvailableFromCurrent(head *sequence, start, curr, end uint64) (uint64, uint64, error) {
+	var bytePos, bitPos uint64
+	if curr != 0 && curr > start {


is it necessary curr != 0? is it not the same as the starting case where start == 0?

but start need not be 0 . but curr could be if not allocated yet

fcrisciani · 2017-06-05T16:41:30Z

bitseq/sequence.go

+		bytePos, bitPos, _ = getFirstAvailable(head, curr)
+		ret := posToOrdinal(bytePos, bitPos)
+		if end < ret {
+			goto begin


you don't really need a goto here.
If start < ret < end then the result is valid and you can return the result. Else you continue you just continue with the other lookup

avoding multiple condition in if. IMO this would be cleaner.

fcrisciani · 2017-06-05T16:45:57Z

bitseq/sequence_test.go

@@ -939,9 +941,6 @@ func TestAllocateRandomDeallocate(t *testing.T) {
 	if hnd.Unselected() != uint64(numBits/2) {
 		t.Fatalf("Expected half sequence. Instead found %d free bits.\nSeed: %d\n%s", hnd.unselected, seed, hnd)
 	}
-	if !hnd.head.equal(expected) {


Is this a check that we may still want to have to validate the consistency of the allocation?

Allocation not be part of the head. So this verification will not hold true with new change.

fcrisciani · 2017-06-05T16:51:04Z

idm/idm_test.go

@@ -110,7 +110,7 @@ func TestUninitialized(t *testing.T) {
 }

 func TestAllocateInRange(t *testing.T) {
-	i, err := New(nil, "myset", 5, 10)
+	i, err := New(nil, "myset", 5, 12)


is there a reason to change this test?

yes since the allocation is serial and not the first available expanding the range to suit the test

fcrisciani · 2017-06-05T17:09:06Z

bitseq/sequence.go

 	for current != nil {
 		if current.block != blockMAX {
 			bytePos, bitPos, err := current.getAvailableBit(bitOffset)
 			return byteOffset + bytePos, bitPos, err
 		}
 		// Moving to next block: Reset bit offset.
 		bitOffset = 0
-		byteOffset += current.count * blockBytes
+		byteOffset += (current.count * blockBytes) - firstOffset


so is this because the current.count of the first byte is not 0?

its because the offset is already calculated and we readd the whole node block count here. This is effective only in scenario where the start is not the from 0 byte and its part of the head RLE node

sorry maybe I'm missing something but if it is part of the head, the current.count should not equal to 0 and so the byteOffset is not modified by the original code?

trapier · 2017-09-21T20:15:00Z

Have not reviewed the code, but very much like the concept. This will provide the gossip control plane maximum time to settle during task churn.

fcrisciani · 2017-09-21T20:18:13Z

@trapier gossip control plane is a transportation mean, does not take any decision on IP matter also the key of every element is the endpoint ID that is unique.
The underlying issue that brings potential IP overlap is due to how swarmkit handles resources, allowing the IP reuse also when a task is in the process to shutdown but still active and running

trapier · 2017-09-22T15:56:33Z

Understood @fcrisciani. What I was thinking is if there are many joins and leaves occurring at the same time on a particular network (and despite the fact that it has a lamport clock), numerically ordered ip allocation will reduce the likelihood of same-time same-address events on the control plane.

fcrisciani · 2017-09-22T15:58:53Z

@trapier sure, but the problem would be in the application layer, because from a networkdb point of view each single event will have a different endpoint ID so a different key and no collision would be possible

mavenugo

Minor Comment. LGTM otherwise.

Can you pls address all other open comments ?

mavenugo · 2017-09-28T09:26:52Z

ipamapi/labels.go

+
+	// AllocSerialPrefix constant marks the reserved label space for libnetwork ipam
+	// allocation ordering.(serial/first available)
+	AllocSerialPrefix = Prefix + ".serial"


can you pls make this .ipam.serial

mavenugo · 2017-09-29T10:48:22Z

@abhi ping

@fcrisciani are you good with this change ?

Previously the bitseq alloc was allocating the first available bit from the begining of the sequence. With this commit the bitseq alloc will proceed from the current allocation. This change will affect the way ipam and vni allocation is done currently. The ip allocation will be done sequentially from the previous allocation as opposed to the first available IP. Signed-off-by: Abhinandan Prativadi <abhi@docker.com>

Since bit allocation is no longer first available from the start some verfications are removed/modified to the change allocation model Signed-off-by: Abhinandan Prativadi <abhi@docker.com>

codecov-io · 2017-09-29T18:50:55Z

Codecov Report

❗ No coverage uploaded for pull request base (master@7447e54). Click here to learn what that means.
The diff coverage is 97.77%.

@@            Coverage Diff            @@
##             master    #1788   +/-   ##
=========================================
  Coverage          ?   38.02%           
=========================================
  Files             ?      137           
  Lines             ?    27313           
  Branches          ?        0           
=========================================
  Hits              ?    10386           
  Misses            ?    15656           
  Partials          ?     1271

Impacted Files	Coverage Δ
drivers/overlay/ov_network.go	`0.25% <0%> (ø)`
bitseq/store.go	`64.89% <100%> (ø)`
ipam/allocator.go	`69.33% <100%> (ø)`
bitseq/sequence.go	`81.83% <100%> (ø)`
idm/idm.go	`76.31% <100%> (ø)`
drivers/overlay/ovmanager/ovmanager.go	`52.34% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7447e54...a2bcac0. Read the comment docs.

fcrisciani · 2017-09-29T19:46:50Z

@mavenugo will give another pass within the eod

fcrisciani · 2017-09-29T21:29:11Z

idm/idm.go

@@ -38,7 +38,7 @@ func (i *Idm) GetID() (uint64, error) {
 	if i.handle == nil {
 		return 0, errors.New("ID set is not initialized")
 	}
-	ordinal, err := i.handle.SetAny()
+	ordinal, err := i.handle.SetAny(false)


don't we need the same sequential allocation for the vxlan? else we are going to hit the vxlan interface file already exists correct?

fcrisciani · 2017-09-29T21:31:01Z

ipamapi/labels.go

+
+	// AllocSerialPrefix constant marks the reserved label space for libnetwork ipam
+	// allocation ordering.(serial/first available)
+	AllocSerialPrefix = Prefix + "ipam.serial"


don't we need a point in the middle?
com.docker.network . ipam.serial?

GordonTheTurtle · 2017-10-03T19:07:00Z

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "ipam_alloc" git@github.com:abhi/libnetwork.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354211816
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

Signed-off-by: Abhinandan Prativadi <abhi@docker.com>

fcrisciani · 2017-10-03T19:32:45Z

idm/idm.go

@@ -56,7 +56,7 @@ func (i *Idm) GetSpecificID(id uint64) error {
 }

 // GetIDInRange returns the first available id in the set within a [start,end] range


I don't think that this is anymore accurate, correct? (being a comment it's not critical, can be handled on top)

fcrisciani · 2017-10-03T19:33:54Z

idm/idm_test.go

+
+	i.Release(52)
+	err = i.GetSpecificID(52)
+	if err != nil {


can you check that is also 52 the value?

fcrisciani · 2017-10-03T19:35:44Z

2 small comments, rest LGTM
@mavenugo PTAL

Backport Serializing bitseq alloc (#1788)

abhi changed the title ~~Serializing bitseq alloc~~ [Not Ready for Review ] Serializing bitseq alloc May 31, 2017

abhi force-pushed the ipam_alloc branch 4 times, most recently from c043f63 to 62472f3 Compare June 5, 2017 05:33

abhi changed the title ~~[Not Ready for Review ] Serializing bitseq alloc~~ Serializing bitseq alloc Jun 5, 2017

fcrisciani reviewed Jun 5, 2017

View reviewed changes

abhi force-pushed the ipam_alloc branch 2 times, most recently from 6180b85 to 61b4076 Compare June 9, 2017 18:58

abhi force-pushed the ipam_alloc branch from 61b4076 to 0223879 Compare July 10, 2017 18:31

abhi force-pushed the ipam_alloc branch from 0223879 to 46f251e Compare July 28, 2017 23:12

abhi force-pushed the ipam_alloc branch 2 times, most recently from 74740ea to 2ce82e5 Compare August 20, 2017 20:11

fcrisciani mentioned this pull request Sep 21, 2017

Docker should avoid re-using IPs too quickly #1945

Closed

mavenugo approved these changes Sep 28, 2017

View reviewed changes

abhi force-pushed the ipam_alloc branch from 2ce82e5 to 8246a51 Compare September 29, 2017 18:21

abhi added 2 commits September 29, 2017 11:33

Minor unit test change

0184de8

Since bit allocation is no longer first available from the start some verfications are removed/modified to the change allocation model Signed-off-by: Abhinandan Prativadi <abhi@docker.com>

abhi force-pushed the ipam_alloc branch from 8246a51 to a650b40 Compare September 29, 2017 18:33

fcrisciani reviewed Sep 29, 2017

View reviewed changes

abhi force-pushed the ipam_alloc branch from a650b40 to 89ebed2 Compare October 3, 2017 19:06

GordonTheTurtle added the dco/no label Oct 3, 2017

Adding a unit case to verify rollover

a2bcac0

Signed-off-by: Abhinandan Prativadi <abhi@docker.com>

abhi force-pushed the ipam_alloc branch from 89ebed2 to a2bcac0 Compare October 3, 2017 19:15

GordonTheTurtle removed the dco/no label Oct 3, 2017

fcrisciani reviewed Oct 3, 2017

View reviewed changes

idm/idm_test.go

i.Release(52)

err = i.GetSpecificID(52)

if err != nil {

Copy link

fcrisciani Oct 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you check that is also 52 the value?

mavenugo approved these changes Oct 3, 2017

View reviewed changes

fcrisciani merged commit 2154459 into moby:master Oct 3, 2017

mavenugo added a commit that referenced this pull request Oct 4, 2017

Merge pull request #1967 from abhi/pick1788

abace68

Backport Serializing bitseq alloc (#1788)

andrewhsu mentioned this pull request Oct 11, 2017

[17.10] re-vndr libnetwork to 6c51292 and swarmkit to 872861d docker-archive/docker-ce#264

Merged

adamancini mentioned this pull request Nov 7, 2017

17.06.2-ee-4 note Serialize IP allocation docker/libnetwork#1788 links to wrong repo docker/docs#5235

Closed

thaJeztah mentioned this pull request Nov 20, 2017

[17.09] Serializing bitseq alloc #2018

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serializing bitseq alloc #1788

Serializing bitseq alloc #1788

abhi commented May 31, 2017 •

edited

Loading

fcrisciani Jun 5, 2017

abhi Jun 5, 2017

fcrisciani Jun 5, 2017

fcrisciani Jun 5, 2017

abhi Jun 5, 2017

fcrisciani Jun 5, 2017

abhi Jun 5, 2017

fcrisciani Jun 5, 2017

abhi Jun 5, 2017

fcrisciani Jun 5, 2017

abhi Jun 5, 2017

fcrisciani Jun 5, 2017

abhi Jun 5, 2017

fcrisciani Jun 5, 2017

abhi Jun 5, 2017

fcrisciani Jun 8, 2017

trapier commented Sep 21, 2017

fcrisciani commented Sep 21, 2017

trapier commented Sep 22, 2017 •

edited

Loading

fcrisciani commented Sep 22, 2017

mavenugo left a comment

mavenugo Sep 28, 2017

mavenugo commented Sep 29, 2017

codecov-io commented Sep 29, 2017 •

edited

Loading

fcrisciani commented Sep 29, 2017

fcrisciani Sep 29, 2017

fcrisciani Sep 29, 2017 •

edited

Loading

GordonTheTurtle commented Oct 3, 2017

fcrisciani Oct 3, 2017

fcrisciani Oct 3, 2017

fcrisciani commented Oct 3, 2017

		@@ -56,7 +56,7 @@ func (i *Idm) GetSpecificID(id uint64) error {
		}

		// GetIDInRange returns the first available id in the set within a [start,end] range

Serializing bitseq alloc #1788

Serializing bitseq alloc #1788

Conversation

abhi commented May 31, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trapier commented Sep 21, 2017

fcrisciani commented Sep 21, 2017

trapier commented Sep 22, 2017 • edited Loading

fcrisciani commented Sep 22, 2017

mavenugo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mavenugo commented Sep 29, 2017

codecov-io commented Sep 29, 2017 • edited Loading

Codecov Report

fcrisciani commented Sep 29, 2017

Choose a reason for hiding this comment

fcrisciani Sep 29, 2017 • edited Loading

Choose a reason for hiding this comment

GordonTheTurtle commented Oct 3, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fcrisciani commented Oct 3, 2017

abhi commented May 31, 2017 •

edited

Loading

trapier commented Sep 22, 2017 •

edited

Loading

codecov-io commented Sep 29, 2017 •

edited

Loading

fcrisciani Sep 29, 2017 •

edited

Loading