Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Encoding incompatibility: operation UTF-16 #972

Open
TiredOfProgramming opened this issue Apr 19, 2018 · 11 comments
Open

Encoding incompatibility: operation UTF-16 #972

TiredOfProgramming opened this issue Apr 19, 2018 · 11 comments

Comments

@TiredOfProgramming
Copy link

TiredOfProgramming commented Apr 19, 2018 •
edited

arachni version: 1.5.1 (ruby 2.2.1p85) [x64-mingw32]
ruby version: 2.5.1p57 (2018-03-29 revision 63029) [x64-mingw32]
windows version: 10 Enterprise

arachni configuration:

OPTIONS:
---
audit:
  parameter_values: true
  exclude_vector_patterns: []
  include_vector_patterns: []
  link_templates: []
  forms: true
  jsons: true
  links: true
  ui_forms: true
  ui_inputs: true
  xmls: true
browser_cluster:
  local_storage: {}
  wait_for_elements: {}
  pool_size: 3
  job_timeout: 30
  worker_time_to_live: 100
  ignore_images: true
  screen_width: 1600
  screen_height: 1200
datastore:
  report_path: \Arachni\arachni_reports\MyApplication\report.afr
http:
  user_agent: PhantomJS / arachni v1.5.1-0.5.12-windows-x86_64
  request_timeout: 300000
  request_redirect_limit: 3
  request_concurrency: 5
  request_queue_size: 300
  request_headers: {}
  response_max_size: 600000
  cookies: {}
  authentication_type: auto
input:
  values: {}
  default_values:
    name: arachni_name
    user: arachni_user
    usr: arachni_user
    pass: 5543!%arachni_secret
    txt: arachni_text
    num: '132'
    amount: '100'
    mail: arachni@email.gr
    account: '12'
    id: '1'
  without_defaults: false
  force: true
scope:
  redundant_path_patterns: {}
  dom_depth_limit: 5
  exclude_file_extensions: []
  exclude_path_patterns:
  - ".png|.ico|.pdf|.jpg|.gif"
  exclude_content_patterns: []
  include_path_patterns: []
  restrict_paths: []
  extend_paths: []
  url_rewrites: {}
  auto_redundant_paths: 5
  directory_depth_limit: 3
  exclude_binaries: true
session:
  check_url: http://servername/example_express/authenticated-page.aspx
  check_pattern: "(?-mix:Welcome user)"
checks:
- trainer
platforms: []
plugins:
  login_script:
    script: \Arachni\arachni_scans\MyApplication\LoginScript.rb
no_fingerprinting: false
authorized_by: john.doe@yahoo.com
url: http://servername/example_express/

Login script below:

browser.goto 'http://servername/example_express/'

sleep 3

begin
browser.text_field(:id => 'username').wait_until_present
puts 'User successfully navigates to the home page'
rescue
puts 'Unable to reach home page'
end

sleep 3

if(browser.url=='http://servername/example_express/')
puts 'Correct URL has been accessed for the Login page'
end

begin
browser.text_field(:id => 'username').wait_until_present  
puts 'Found User Name text field'
rescue
puts 'Unable to locate User Name text field'
end

begin
browser.text_field(:id => 'password').wait_until_present		
puts 'Found Password text field'
rescue
puts 'Unable to locate Password text field'
end

browser.text_field(:id => 'username').set 'username@aol.com'
puts 'Was able to successfully set username'

browser.text_field(:id => 'password').set 'password'
puts 'Was able to successfully set the password'

browser.button(:id => 'loginButton').click
puts 'Was able to click on Login button'

sleep 3

begin
browser.a(:id => 'logoutLink').wait_until_present
puts "User is successfully lands on the profile page with the URL: " + browser.url
rescue
puts "Login failed"
end

framework.options.session.check_url = browser.url
framework.options.session.check_pattern = /Welcome user/

Error stack trace:

[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] [Encoding::CompatibilityError] incompatible encoding with this operation: UTF-16
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser/nodes/with_value.rb:22:in `strip'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser/nodes/with_value.rb:22:in `value='
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser/nodes/with_value.rb:18:in `initialize'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser/sax.rb:58:in `new'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser/sax.rb:58:in `text'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser.rb:70:in `sax_html'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser.rb:70:in `block in parse'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/support/cache/base.rb:108:in `call'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/support/cache/base.rb:108:in `fetch'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser.rb:66:in `parse'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser.rb:243:in `document'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/parser.rb:423:in `paths'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/page.rb:309:in `paths'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework/parts/data.rb:207:in `push_paths_from_page'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:98:in `audit_page'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:228:in `audit_queues'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:202:in `block in audit'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:177:in `loop'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework/parts/audit.rb:177:in `audit'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework.rb:117:in `block in run'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/utilities.rb:425:in `call'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/utilities.rb:425:in `exception_jail'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework.rb:117:in `run'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:428] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/ui/cli/framework.rb:80:in `block in run'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:429] 
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:430] Parent:
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:431] Arachni::Framework
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:432] 
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:433] Block:
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:434] #<Proc:0x00000007c865c0@~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework.rb:117>
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:435] 
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:436] Caller:
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:437] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/utilities.rb:425:in `exception_jail'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:437] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/lib/arachni/framework.rb:117:in `run'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:437] ~/arachni-1.5.1-0.5.12-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/gems/arachni-1.5.1/ui/cli/framework.rb:80:in `block in run'
[2018-04-19 15:30:12 -0400] [utilities#exception_jail:438] --------------------------------------------------------------------------------

Steps are:

  1. User navigates to http://servername/example_express/
  2. User clicks on Login button and navigates to http://servername/example_express/login.aspx
  3. User supplies username and password, clicks on Login button and navigates to http://servername/example_express/authenticated-page.aspx

I'm using login_script for logging in. CLI displays messages that Login was successful and also displays cookies assigned. Right after that the scan is being aborted with the aforementioned error message. Please advise if this is a bug in the arachni framework and how this can be overcome.
@TiredOfProgramming
Copy link
Author

@Zapotek Were you able to review this issue?

@Zapotek
Copy link
Member

Zapotek commented Apr 25, 2018

Can you please give the nightlies a try and see if the issue is still there?

@TiredOfProgramming
Copy link
Author

Which nighlies would you recommend? Will you be able to provide link please?

@Zapotek
Copy link
Member

Zapotek commented Apr 25, 2018

http://downloads.arachni-scanner.com/nightlies/

@TiredOfProgramming
Copy link
Author

@Zapotek So I extracted nighlies build to the different folder named arachni-2.0. The final destination of the bin folder is C:\JohnDoe\arachni-2.0\arachni-2.0dev-1.0dev-windows-x86_64\arachni-2.0dev-1.0dev-windows-x86_64\bin Simply running arachni --version command leads to following error message C:/JohnDoe/arachni-2.0/arachni-2.0dev-1.0dev-windows-x86_64/arachni-2.0dev-1.0dev-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/bundler/gems/arachni-dfdc70df98a8/lib/arachni/parser/nodes/element/with_attributes.rb:9:in require_relative: cannot load such file -- C:/JohnDoe/arachni-2.0/arachni-2.0dev-1.0dev-windows-x86_64/arachni-2.0dev-1.0dev-windows-x86_64/system/ruby/lib/ruby/gems/2.2.0/bundler/gems/arachni-dfdc70df98a8/lib/arachni/parser/nodes/element/with_attributes/attributes (LoadError)

Sorry man, I tried 😃 Do you have an idea of what might be wrong in here? As reminder: I'm running on Windows 10 Enterprise.

@TiredOfProgramming
Copy link
Author

@Zapotek Were you able to look into the issue? Thanks for support

@Zapotek
Copy link
Member

Zapotek commented May 1, 2018

Does the path happen to have special characters or spaces or things like that?
If so try extracting in a different directory and see if that helps.

@TiredOfProgramming
Copy link
Author

TiredOfProgramming commented May 1, 2018 •
edited

@Zapotek Thanks for the response. The path that fails with the message does contains dashes (-) like: http://www.example.com/welcome-authenticated-user.aspx Not sure if this can't be parsed with arachni default encoder UTF-8. I've arachni crawled bunch of other different web applications with the dashes ( - ) and underscores ( _ ) in the paths without any problems.

What do you mean try extract to different directory? Can you please clarify a little bit?

@Zapotek
Copy link
Member

Zapotek commented May 1, 2018

I meant your local path where you extracted the nightly.

@TiredOfProgramming
Copy link
Author

Alright, I got it. I thought you were referring to the issue with Encoding Incompatibility itself, not realizing that regarding nightly build. Let me see and I'll get back with the results. Thanks!

@TiredOfProgramming
Copy link
Author

@Zapotek I verified that local path doesn't contain any special characters, and run the scan again against the app that was failing with Incompatibility issue with the original (stable version) arachni.

Were you able to review if the change for incompatible encoding can be merged with the main branch? Thanks.

@easycloudcompany easycloudcompany mentioned this issue Feb 8, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Zapotek @TiredOfProgramming