Skip to content

Custom exit code for fallback mode error#284

Merged
3v0k4 merged 3 commits intomasterfrom
fallback-error
Nov 11, 2024
Merged

Custom exit code for fallback mode error#284
3v0k4 merged 3 commits intomasterfrom
fallback-error

Conversation

@3v0k4
Copy link
Copy Markdown
Contributor

@3v0k4 3v0k4 commented Nov 11, 2024

Story

https://trello.com/c/GCRO05YP/513-custom-exit-code-when-cannot-connect-to-the-api-and-fallback-mode-is-disabled

Description

Allow users to set a custom error exit code (KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE) whenever Knapsack Pro fails because Fallback Mode cannot be used.

Since we are catching the error, we don't print the backtraces anymore. This is a plus as you'd expect those only in case of an unhandled error. See below for the actual differences.

Checklist reminder

  • You added the changes to the UNRELEASED section of the CHANGELOG.md, including the needed bump (ie, patch, minor, major)
  • You follow the architecture outlined below for RSpec in Queue Mode, which is a work in progress (feel free to propose changes):
    • Pure: lib/knapsack_pro/pure/queue/rspec_pure.rb contains pure functions that are unit tested.
    • Extension: lib/knapsack_pro/extensions/rspec_extension.rb encapsulates calls to RSpec internals and is integration and e2e tested.
    • Runner: lib/knapsack_pro/runners/queue/rspec_runner.rb invokes the pure code and the extension to produce side effects, which are integration and e2e tested.

Comparison before/after

I run locally Knapsack Pro with:

  KNAPSACK_PRO_ENDPOINT=http://TODOapi.knapsackpro.test:3000 \
  KNAPSACK_PRO_MAX_REQUEST_RETRIES=0 \
  KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false \

And additionally (to check the other failure path):

  KNAPSACK_PRO_FALLBACK_MODE_ENABLED=true \
  KNAPSACK_PRO_CI_NODE_RETRY_COUNT=1 \

bin/knapsack_pro_rspec; echo $?

after

W, [2024-11-08T16:40:34.478942 #90791]  WARN -- : [knapsack_pro] POST http://TODOapi.knapsackpro.test:3000/v1/build_distributions/subset
W, [2024-11-08T16:40:34.479047 #90791]  WARN -- : [knapsack_pro] Request failed due to:
W, [2024-11-08T16:40:34.479080 #90791]  WARN -- : [knapsack_pro] #<Errno::ECONNREFUSED: Failed to open TCP connection to TODOapi.knapsackpro.test:3000 (Connection refused - connect(2) for "TODOapi.knapsackpro.test" port 3000)>
E, [2024-11-08T16:40:34.479153 #90791] ERROR -- : [knapsack_pro] Fallback Mode was disabled with KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Please restart this CI node to retry tests. Most likely Fallback Mode was disabled due to https://knapsackpro.com/perma/ruby/regular-mode-connection-error-with-fallback-enabled-false
1

before

W, [2024-11-08T16:44:33.584195 #90992]  WARN -- : [knapsack_pro] POST http://TODOapi.knapsackpro.test:3000/v1/build_distributions/subset
W, [2024-11-08T16:44:33.584308 #90992]  WARN -- : [knapsack_pro] Request failed due to:
W, [2024-11-08T16:44:33.584334 #90992]  WARN -- : [knapsack_pro] #<Errno::ECONNREFUSED: Failed to open TCP connection to TODOapi.knapsackpro.test:3000 (Connection refused - connect(2) for "TODOapi.knapsackpro.test" port 3000)>
E, [2024-11-08T16:44:33.584396 #90992] ERROR -- : [knapsack_pro] Fallback Mode was disabled with KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Please restart this CI node to retry tests. Most likely Fallback Mode was disabled due to https://knapsackpro.com/perma/ruby/regular-mode-connection-error-with-fallback-enabled-false
rake aborted!
Fallback Mode was disabled with KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Please restart this CI node to retry tests. Most likely Fallback Mode was disabled due to https://knapsackpro.com/perma/ruby/regular-mode-connection-error-with-fallback-enabled-false
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/allocator.rb:32:in `test_file_paths'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/base_runner.rb:16:in `test_file_paths'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/base_runner.rb:28:in `test_files_to_execute_exist?'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/rspec_runner.rb:16:in `run'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/tasks/rspec.rake:7:in `block (2 levels) in <top (required)>'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/bin/ruby_executable_hooks:22:in `eval'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/bin/ruby_executable_hooks:22:in `<main>'
Tasks: TOP => knapsack_pro:rspec
(See full trace by running task with --trace)
1

bin/knapsack_pro_queue_rspec; echo $?

after

I, [2024-11-08T16:40:55.104101 #90819]  INFO -- : [knapsack_pro] KNAPSACK_PRO_FIXED_QUEUE_SPLIT is not set. Using default value: true. Learn more at https://knapsackpro.com/perma/ruby/fixed-queue-split
W, [2024-11-08T16:40:55.109043 #90819]  WARN -- : [knapsack_pro] POST http://TODOapi.knapsackpro.test:3000/v1/queues/queue
W, [2024-11-08T16:40:55.109099 #90819]  WARN -- : [knapsack_pro] Request failed due to:
W, [2024-11-08T16:40:55.109132 #90819]  WARN -- : [knapsack_pro] #<Errno::ECONNREFUSED: Failed to open TCP connection to TODOapi.knapsackpro.test:3000 (Connection refused - connect(2) for "TODOapi.knapsackpro.test" port 3000)>
E, [2024-11-08T16:40:55.109230 #90819] ERROR -- : [knapsack_pro] Fallback Mode was disabled with KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Please restart this CI node to retry tests. Most likely Fallback Mode was disabled due to https://knapsackpro.com/perma/ruby/queue-mode-connection-error-with-fallback-enabled-false

Finished in 0.08829 seconds (files took 2.85 seconds to load)
0 examples, 0 failures

1

It's a bit unfortunate that we print the summary as it seems nothing went wrong. But I think it's not that bad.

before

I, [2024-11-08T16:44:55.087071 #91021]  INFO -- : [knapsack_pro] KNAPSACK_PRO_FIXED_QUEUE_SPLIT is not set. Using default value: true. Learn more at https://knapsackpro.com/perma/ruby/fixed-queue-split
W, [2024-11-08T16:44:55.091751 #91021]  WARN -- : [knapsack_pro] POST http://TODOapi.knapsackpro.test:3000/v1/queues/queue
W, [2024-11-08T16:44:55.091798 #91021]  WARN -- : [knapsack_pro] Request failed due to:
W, [2024-11-08T16:44:55.091821 #91021]  WARN -- : [knapsack_pro] #<Errno::ECONNREFUSED: Failed to open TCP connection to TODOapi.knapsackpro.test:3000 (Connection refused - connect(2) for "TODOapi.knapsackpro.test" port 3000)>
E, [2024-11-08T16:44:55.091895 #91021] ERROR -- : [knapsack_pro] Fallback Mode was disabled with KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Please restart this CI node to retry tests. Most likely Fallback Mode was disabled due to https://knapsackpro.com/perma/ruby/queue-mode-connection-error-with-fallback-enabled-false

Finished in 0.08774 seconds (files took 2.71 seconds to load)
0 examples, 0 failures

E, [2024-11-08T16:44:55.094315 #91021] ERROR -- : [knapsack_pro] An unexpected exception happened. RSpec cannot handle it. The exception: #<RuntimeError: Fallback Mode was disabled with KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Please restart this CI node to retry tests. Most likely Fallback Mode was disabled due to https://knapsackpro.com/perma/ruby/queue-mode-connection-error-with-fallback-enabled-false>
E, [2024-11-08T16:44:55.094358 #91021] ERROR -- : [knapsack_pro] Exception message: Fallback Mode was disabled with KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false. Please restart this CI node to retry tests. Most likely Fallback Mode was disabled due to https://knapsackpro.com/perma/ruby/queue-mode-connection-error-with-fallback-enabled-false
E, [2024-11-08T16:44:55.094463 #91021] ERROR -- : [knapsack_pro] Exception backtrace: /Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/queue_allocator.rb:34:in `test_file_paths'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/queue/base_runner.rb:29:in `test_file_paths'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb:153:in `pull_tests_from_queue'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb:78:in `block in with_batch'
<internal:kernel>:187:in `loop'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb:76:in `with_batch'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/extensions/rspec_extension.rb:93:in `block (2 levels) in knapsack__run_specs'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2091:in `with_suite_hooks'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/extensions/rspec_extension.rb:92:in `block in knapsack__run_specs'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rspec-core-3.13.0/lib/rspec/core/reporter.rb:74:in `report'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/extensions/rspec_extension.rb:91:in `knapsack__run_specs'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb:54:in `run'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb:20:in `run'
/Users/riccardoodone/code/knapsack/knapsack_pro-ruby/lib/tasks/queue/rspec.rake:12:in `block (3 levels) in <top (required)>'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/task.rb:279:in `block in execute'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/task.rb:279:in `each'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/task.rb:279:in `execute'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/task.rb:199:in `synchronize'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/task.rb:188:in `invoke'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:188:in `invoke_task'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:138:in `each'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:138:in `block in top_level'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:147:in `run_with_threads'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:132:in `top_level'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:83:in `block in run'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:214:in `standard_exception_handling'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/lib/rake/application.rb:80:in `run'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/bin/rake:25:in `load'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/bin/rake:25:in `<main>'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/bin/ruby_executable_hooks:22:in `eval'
/Users/riccardoodone/.rvm/gems/ruby-3.3.5/bin/ruby_executable_hooks:22:in `<main>'

1

@3v0k4 3v0k4 force-pushed the fallback-error branch 2 times, most recently from 5ae80e4 to 8e1e627 Compare November 11, 2024 10:30
@3v0k4 3v0k4 self-assigned this Nov 11, 2024
@3v0k4 3v0k4 marked this pull request as ready for review November 11, 2024 11:36
@ArturT
Copy link
Copy Markdown
Member

ArturT commented Nov 11, 2024

#suggestion

Ensure the new feature works in a real project scenario just in case.

# .circleci/config.yml

      - run:
          working_directory: ~/rails-app-with-knapsack_pro
          command: |
            # fallback ||
            export KNAPSACK_PRO_ENDPOINT=https://api-fake.knapsackpro.com
            export KNAPSACK_PRO_MAX_REQUEST_RETRIES=1
            bundle exec rake knapsack_pro:rspec

      # add the following
      - run:
          working_directory: ~/rails-app-with-knapsack_pro
          command: |
            # fallback off and custom exit code ||
            export KNAPSACK_PRO_ENDPOINT=https://api-fake.knapsackpro.com
            export KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false
            export KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE=0 # won't fail the build and it won't run tests
            bundle exec rake knapsack_pro:rspec

...

# queue


      - run:
          working_directory: ~/rails-app-with-knapsack_pro
          command: |
            # fallback ||
            export KNAPSACK_PRO_ENDPOINT=https://api-fake.knapsackpro.com
            export KNAPSACK_PRO_MAX_REQUEST_RETRIES=1
            bundle exec rake knapsack_pro:queue:rspec

      # add the following
      - run:
          working_directory: ~/rails-app-with-knapsack_pro
          command: |
            # fallback off and custom exit code ||
            export KNAPSACK_PRO_ENDPOINT=https://api-fake.knapsackpro.com
            export KNAPSACK_PRO_FALLBACK_MODE_ENABLED=false
            export KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE=0 # won't fail the build and it won't run tests
            bundle exec rake knapsack_pro:queue:rspec

Comment thread lib/knapsack_pro/config/env.rb Outdated
end

def fallback_mode_error_exit_code_or(default)
env_value = ENV['KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE']
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion

I would do this:

Suggested change
env_value = ENV['KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE']
ENV.fetch('KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE', 1).to_i

I would add a unit test in spec/knapsack_pro/config/env_spec.rb for KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE to explain there that 1 means the default exit code when KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE is not defined.

I would rename fallback_mode_error_exit_code_or to fallback_mode_error_exit_code.

No need to define 1 as default exit code each time you call this method in multiple test runners. By default 1 is exit code in case of an error.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would add a unit test in spec/knapsack_pro/config/env_spec.rb for KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE to explain there that 1 means the default exit code when KNAPSACK_PRO_FALLBACK_MODE_ERROR_EXIT_CODE is not defined.

I think it's clear enough by looking at the code, no need to add test cases.

I would rename fallback_mode_error_exit_code_or to fallback_mode_error_exit_code.
No need to define 1 as default exit code each time you call this method in multiple test runners. By default 1 is exit code in case of an error.

Great point. Applied.

@3v0k4
Copy link
Copy Markdown
Contributor Author

3v0k4 commented Nov 11, 2024

Ensure the new feature works in a real project scenario just in case.

I decided not to test it e2e because:

  • it's already covered at the integration level
  • it's a marginal feature
  • the proposed tests is a bit hacky

I'd prefer to keep the e2e tests as smoke tests that don't cover edge cases.

@3v0k4 3v0k4 merged commit 1adc442 into master Nov 11, 2024
@3v0k4 3v0k4 deleted the fallback-error branch November 11, 2024 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants