diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index e94a510..d6f11ff 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -34,7 +34,7 @@ jobs: - name: Run benchmark tests run: bundle exec ruby benchmark.rb env: - WORKFLOW_PORT: 30001 + REDIS_PORT: 30001 - name: Stop redis cluster run: bin/cluster stop diff --git a/.rubocop.yml b/.rubocop.yml index 633e2d1..8c13b44 100644 --- a/.rubocop.yml +++ b/.rubocop.yml @@ -41,3 +41,7 @@ RSpec/MultipleExpectations: # Example has too many lines. [8/5] RSpec/ExampleLength: Max: 20 + +# Rubygems 2FA can't be used with github publish workflow +Gemspec/RequireMFA: + Enabled: false diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ce0364..48a034b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,37 @@ -## [Unreleased] +## [0.1.3] - 2025-02-08 + +## What's Changed + +**Full Changelog**: https://github.com/lifeBCE/redis-single-file/compare/v0.1.2...v0.1.3 + +--- +
+ +## [0.1.2] - 2025-02-08 + +## What's Changed +* publish workflow by @lifeBCE in https://github.com/lifeBCE/redis-single-file/pull/3 +* IGNORE_VERSION: "true" by @lifeBCE in https://github.com/lifeBCE/redis-single-file/pull/4 +* cluster support - v0.1.2 by @lifeBCE in https://github.com/lifeBCE/redis-single-file/pull/5 + +**Full Changelog**: https://github.com/lifeBCE/redis-single-file/compare/v0.1.1...v0.1.2 + +--- +
+ +## [0.1.1] - 2025-02-03 + +## What's Changed +* github workflows by @lifeBCE in https://github.com/lifeBCE/redis-single-file/pull/1 +* badges by @lifeBCE in https://github.com/lifeBCE/redis-single-file/pull/2 + +## New Contributors +* @lifeBCE made their first contribution in https://github.com/lifeBCE/redis-single-file/pull/1 + +**Full Changelog**: https://github.com/lifeBCE/redis-single-file/commits/v0.1.1 + +--- +
## [0.1.0] - 2025-01-31 diff --git a/Gemfile.lock b/Gemfile.lock index 8bc82b6..5999998 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ PATH remote: . specs: - redis-single-file (0.1.2) + redis-single-file (0.1.3) redis (~> 5.3.0) redis-clustering (~> 5.3.0) diff --git a/README.md b/README.md index 7cce004..e1bd8ab 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,7 @@ [![Build Status](https://github.com/lifeBCE/redis-single-file/actions/workflows/build.yml/badge.svg)](https://github.com/lifeBCE/redis-single-file/actions/workflows/build.yml) [![RSpec Status](https://github.com/lifeBCE/redis-single-file/actions/workflows/rspec.yml/badge.svg)](https://github.com/lifeBCE/redis-single-file/actions/workflows/rspec.yml) -[![CodeQL Status](https://github.com/lifeBCE/redis-single-file/actions/workflows/codeql.yml/badge.svg)](https://github.com/lifeBCE/redis-single-file/actions/workflows/codeql.yml) [![Rubocop Status](https://github.com/lifeBCE/redis-single-file/actions/workflows/rubocop.yml/badge.svg)](https://github.com/lifeBCE/redis-single-file/actions/workflows/rubocop.yml) -[![Benchmark Status](https://github.com/lifeBCE/redis-single-file/actions/workflows/benchmark.yml/badge.svg)](https://github.com/lifeBCE/redis-single-file/actions/workflows/benchmark.yml) +[![CodeQL Status](https://github.com/lifeBCE/redis-single-file/actions/workflows/codeql.yml/badge.svg)](https://github.com/lifeBCE/redis-single-file/actions/workflows/codeql.yml) # Redis Single File - Distributed Execution Synchronization @@ -79,27 +78,29 @@ end ### Distributed Queue Design -The redis `blpop` command will attempt to pop (delete and return) a value from -a queue but will block when no values are present in the queue. A timeout can -be provided to prevent deadlock situations. - -To unblock (unlock) an instance, add/push an item to the queue. This is done -one at a time to controll the serialization of the distrubuted execution. Redis -selects the instance waiting the longest each time a new token is added. +> [!IMPORTANT] +> The redis `blpop` command will attempt to pop (delete and return) a value from +> a queue but will block when no values are present in the queue. A timeout can +> be provided to prevent deadlock situations. +> +> To unblock (unlock) an instance, add/push an item to the queue. This is done +> one at a time to controll the serialization of the distrubuted execution. Redis +> selects the instance waiting the longest each time a new token is added. ### Auto Expiration -All redis keys are expired and automatically removed after a certain period -but will be recreated again on the next use. Each new client should face one -of two scenarios when entering synchronization. - -1. The mutex key is not set causing the client to create the keys and prime - the queue with its first token unlocking it for the first execution. - -2. The mutex key is already set so the client will skip the priming and enter - directly into the queue where it should immediately find a token left by - the last client upon completion or block waiting for the current client to - finish execution. +> [!NOTE] +> All redis keys are expired and automatically removed after a certain period +> but will be recreated again on the next use. Each new client should face one +> of two scenarios when entering synchronization. +> +> 1. The mutex key is not set causing the client to create the keys and prime +> the queue with its first token unlocking it for the first execution. +> +> 2. The mutex key is already set so the client will skip the priming and enter +> directly into the queue where it should immediately find a token left by +> the last client upon completion or block waiting for the current client to +> finish execution. ### Considerations over redlock approach @@ -178,6 +179,95 @@ Comparison: forked (10x): 56.6 i/s - 76.90x slower ``` +## Cluster Management + +After installing redis locally, you can use the provided `bin/cluster` script to manage a local cluster. To customize your local cluster, edit the `bin/cluster` script to provide your own values for the following script variables. + +```bash +# +# configurable settings +# +HOST=127.0.0.1 +PORT=30000 +MASTERS=3 # min 3 for cluster +REPLICAS=2 # replicas per master +TIMEOUT=2000 +PROTECTED_MODE=yes +ADDITIONAL_OPTIONS="" +``` + +
+Start cluster nodes + + $ bin/cluster start + +```console +Starting 30001 +Starting 30002 +Starting 30003 +Starting 30004 +Starting 30005 +Starting 30006 +Starting 30007 +Starting 30008 +Starting 30009 +``` +
+ +
+Create cluster configuration + + $ bin/cluster create -f + +```console +>>> Performing hash slots allocation on 9 nodes... +Master[0] -> Slots 0 - 5460 +Master[1] -> Slots 5461 - 10922 +Master[2] -> Slots 10923 - 16383 +Adding replica 127.0.0.1:30005 to 127.0.0.1:30001 +Adding replica 127.0.0.1:30006 to 127.0.0.1:30001 +Adding replica 127.0.0.1:30007 to 127.0.0.1:30002 +Adding replica 127.0.0.1:30008 to 127.0.0.1:30002 +Adding replica 127.0.0.1:30009 to 127.0.0.1:30003 +Adding replica 127.0.0.1:30004 to 127.0.0.1:30003 +``` +
+ +
+Stop cluster nodes + + $ bin/cluster stop + +```console +Stopping 30001 +Stopping 30002 +Stopping 30003 +Stopping 30004 +Stopping 30005 +Stopping 30006 +Stopping 30007 +Stopping 30008 +Stopping 30009 +``` +
+ +
+Clean local cluster files + + $ bin/cluster clean + +```console +Cleaning *.log +Cleaning appendonlydir-* +Cleaning dump-*.rdb +Cleaning nodes-*.conf +``` +
+ +After the cluster is running and configured, you can direct the `test.rb` and `benchmark.rb` scripts at the cluster by setting the port on execution. + + $ REDIS_PORT=30001 bundle exec ruby benchmark.rb + ## Disclaimer > [!WARNING] diff --git a/benchmark.rb b/benchmark.rb index c6ff5a0..aa1bbf6 100644 --- a/benchmark.rb +++ b/benchmark.rb @@ -3,7 +3,7 @@ require 'benchmark/ips' require 'redis_single_file' -PORT = ENV['WORKFLOW_PORT'] || 6379 +PORT = ENV['REDIS_PORT'] || 6379 scenario_1_semaphore = RedisSingleFile.new(name: :scenario1, port: PORT) scenario_2_semaphore = RedisSingleFile.new(name: :scenario2, port: PORT) diff --git a/bin/cluster b/bin/cluster index 360b9eb..2a062d4 100755 --- a/bin/cluster +++ b/bin/cluster @@ -11,6 +11,14 @@ # bin/cluster stop # ... # +# --- +# +# This shell script is an adaptation of the original script from the redis +# gem that can be found at the link below. Appreciate the head start! +# +# Redis create-cluster script: +# github.com/redis/redis/blob/unstable/utils/create-cluster/create-cluster +# # # local run env settings diff --git a/lib/redis_single_file/cluster_client_builder.rb b/lib/redis_single_file/cluster_client_builder.rb index 6bfe3b7..084c820 100644 --- a/lib/redis_single_file/cluster_client_builder.rb +++ b/lib/redis_single_file/cluster_client_builder.rb @@ -15,7 +15,7 @@ class << self # # Delegates class method calls to instance method # - # @param [...] params passes directly to constructor + # @param [...] params passed directly to constructor # @return [Redis::Cluster] redis cluster instance def call(...) = new(...).call end @@ -40,15 +40,21 @@ def initialize(redis:) def call raise ClusterDisabledError, 'cluster not detected' unless cluster_enabled? - # use extracted client options with parsed nodes - Redis::Cluster.new(**client_options, nodes:) + # use extracted client options with parsed master nodes + Redis::Cluster.new(**client_options, nodes: master_nodes) end private # ================================================================== attr_reader :redis - def nodes + def cluster_enabled? + redis.info('cluster')['cluster_enabled'] == '1' + rescue Redis::CommandError + false # assume cluster mode is disabled + end + + def master_nodes cluster_nodes.filter_map do |node| next unless node[:flags].include?('master') @@ -56,10 +62,18 @@ def nodes end end - def cluster_enabled? - redis.info('cluster')['cluster_enabled'] == '1' - rescue Redis::CommandError - false # assume cluster mode is disabled + def client_options + config = redis._client.config + params = %i[ + db ssl host port path custom username password protocol + ssl_params read_timeout write_timeout connect_timeout + ] + + params_hash = params.each.with_object({}) do |key, memo| + memo[key] = config.public_send(key) + end + + params_hash.merge(url: config.server_url) end def cluster_nodes @@ -82,19 +96,5 @@ def cluster_nodes } end end - - def client_options - config = redis._client.config - params = %i[ - db ssl host port path custom username password protocol - ssl_params read_timeout write_timeout connect_timeout - ] - - params_hash = params.each.with_object({}) do |key, memo| - memo[key] = config.public_send(key) - end - - params_hash.merge(url: config.server_url) - end end end diff --git a/lib/redis_single_file/version.rb b/lib/redis_single_file/version.rb index 8705a39..36bb7c8 100644 --- a/lib/redis_single_file/version.rb +++ b/lib/redis_single_file/version.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true module RedisSingleFile - VERSION = '0.1.2' + VERSION = '0.1.3' end diff --git a/redis-single-file.gemspec b/redis-single-file.gemspec index 4386712..c4fc997 100644 --- a/redis-single-file.gemspec +++ b/redis-single-file.gemspec @@ -3,41 +3,50 @@ require_relative 'lib/redis_single_file/version' Gem::Specification.new do |spec| - spec.name = 'redis-single-file' + spec.name = 'redis-single-file' spec.version = RedisSingleFile::VERSION spec.authors = ['LifeBCE'] - spec.email = ['eric06@gmail.com'] + spec.email = ['eric06@gmail.com'] - spec.summary = 'Distributed semaphore implementation with redis.' + spec.summary = 'Distributed semaphore implementation with redis.' spec.description = 'Synchronize execution across numerous instances.' - spec.homepage = 'https://github.com/lifeBCE/redis-single-file' - spec.license = 'MIT' - spec.required_ruby_version = '>= 3.2.0' + spec.homepage = 'https://github.com/lifeBCE/redis-single-file' + spec.license = 'MIT' - # spec.metadata['allowed_push_host'] = "TODO: Set to your gem server 'https://example.com'" + # Build Requiremenrs + spec.require_paths = ['lib'] + spec.required_ruby_version = '>= 3.2.0' spec.metadata['homepage_uri'] = spec.homepage - # spec.metadata["source_code_uri"] = "TODO: Put your gem's public repo URL here." - # spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here." - - # Specify which files should be added to the gem when it is released. - # The `git ls-files -z` loads the files in the RubyGem that have been added into git. - gemspec = File.basename(__FILE__) - spec.files = IO.popen(%w[git ls-files -z], chdir: __dir__, err: IO::NULL) do |ls| - ls.readlines("\x0", chomp: true).reject do |f| - (f == gemspec) || - f.start_with?(*%w[bin/ test/ spec/ features/ .git .github appveyor Gemfile]) + spec.metadata['allowed_push_host'] = 'https://rubygems.org' + spec.metadata['changelog_uri'] = 'https://github.com/lifeBCE/redis-single-file/blob/main/CHANGELOG.md' + + # Local variables used when setting spec.files below + gemspec = File.basename(__FILE__) + rejected = %w[bin/ test/ spec/ features/ .git .github appveyor Gemfile] + + # NOTE: Grabs list of files to include in gem from git. New files/directories + # will not be picked up until running 'git add' prior to building gem. + # + spec.files = + IO.popen(%w[git ls-files -z], chdir: __dir__, err: IO::NULL) do |ls| + # reject self and rejected paths defined above + ls.readlines("\x0", chomp: true).reject do |f| + (f == gemspec) || f.start_with?(*rejected) + end end - end + + # Identify Gem Exectuables spec.bindir = 'exe' spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) } - spec.require_paths = ['lib'] - # Uncomment to register a new dependency of your gem + # Redis Single File Dependencies spec.add_dependency 'redis', '~> 5.3.0' spec.add_dependency 'redis-clustering', '~> 5.3.0' + # Disable MFA Requirement - github publishing can't support + spec.metadata['rubygems_mfa_required'] = 'false' + # For more information and examples about making a new gem, check out our # guide at: https://bundler.io/guides/creating_gem.html - spec.metadata['rubygems_mfa_required'] = 'true' end diff --git a/test.rb b/test.rb index fe6399d..b3b448c 100644 --- a/test.rb +++ b/test.rb @@ -1,15 +1,17 @@ #!/usr/bin/env ruby require 'pry' +require 'securerandom' require 'redis_single_file' -RUN_ID = 'same-same' #SecureRandom.uuid +PORT = ENV['REDIS_PORT'] || 6379 +RUN_ID = SecureRandom.uuid ITERATIONS = (ARGV[0] || 10).to_i WORK_LOAD = (ARGV[1] || 1).to_i TIMEOUT = ITERATIONS * WORK_LOAD -#semaphore = RedisSingleFile.new(name: RUN_ID, port: 30001) +#semaphore = RedisSingleFile.new(name: RUN_ID, port: PORT) #semaphore.synchronize!(timeout: 10) do # puts "Hello World!" # sleep 1 @@ -19,7 +21,7 @@ #10.times.map do # fork do -# semaphore = RedisSingleFile.new(name: RUN_ID) +# semaphore = RedisSingleFile.new(name: RUN_ID, port: PORT) # semaphore.synchronize!(timeout: TIMEOUT) do # puts "Hello World!" # sleep WORK_LOAD @@ -35,7 +37,7 @@ threads = ITERATIONS.times.map do thread = Thread.new do - semaphore = RedisSingleFile.new(name: RUN_ID, port: 30001) + semaphore = RedisSingleFile.new(name: RUN_ID, port: PORT) semaphore.synchronize(timeout: TIMEOUT) do puts "Hello World!" sleep WORK_LOAD