Skip to content
A distributed resource allocator
Java Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
src
.gitignore
.travis.yml
LICENSE
README.md
deploy.sh
perform-release.sh
pom.xml
prep-release.sh
rollback-release.sh
settings.xml

README.md

paradoxical.dalloc

Build status

A distributed resource allocator. It supports

  • Hazelcast distribution
  • Manual distribution

What does it solve?

The problem we are solving is having many distributed machines who should act on only 1 resource out of a set of resources. For example, you have 10,000 items that each need monitoring, and you don't want more than one machine monitoring an element in the set. You need to distribute this work to a cluster of boxes such that each box is monitoring 10,000/clusterSize elements.

Installation

To install

<dependency>
    <groupId>io.paradoxical</groupId>
    <artifactId>dalloc</artifactId>
    <version>1.0</version>
</dependency>

Hazelcast Distribution

Distribution in hazelcast constitutes of mapping your input set of resources to a ResourceIdentifier and grouping them as part of a ResourceGroup. Given the set of members in the hazelcast instance the resources are distributed evenly to each member. As members join the cluster they may rebalance resources (i.e. give up some resources they had so that new members can claim them). As members leave the cluster, members will rebalance the required load.

Claiming resources

Claim event can occur by either

  • A member joining
  • A member leaving
  • Manually invoked

The claim function will be dispatched on member join/leave events and is executed on the same thread. If you need to do long running work during a claim event it is best to capture the Set of claimed resources and dispatch your work on a threadpool.

Split brain

There is no splitbrain merge exposed here.

How it works

There is a single Map<ClusterMember, Set<ResourceIdentifier>> stored on a distributed node (to minimize cluster IO) that each member tries to lock and acquire. When someone acquires a lock they find out the intersection of the total vs already allocated (owned by the set) and divies up the required needing allocation

Needing Acquisition = Total available ∩ Already allocated

When a member leaves, all members will try and acquire the lock and prune that members resources.

Manual distribution

Manual distribution relies on the user to define the total expected set of workers and to manually label each worker. Each worker will then grab

Total Set / Workers

Resources. There is no guarantee of overlap protection, but since resource identifiers are ordered they will deterministically choose the same slice sections.

You can’t perform that action at this time.