Asterisks not encoded properly and result in error 32 "Could not authenticate you."

[jeremyhaile]

Twitter-text (1.11.0) says that tweets with asterisks are valid. And I am also allowed to post them to Twitter directly. However, when posted through the Twitter gem (6.0.0) I receive the error: "Could not authenticate you." (code 32)

Twitter::Validation.tweet_invalid?("*TEST* Does this work?") # returns false (valid tweet)

client.update("*TEST* Does this work?") # raises Twitter::Error::Unauthorized: Could not authenticate you.

client.update("TEST Does this work?") # works

This forum post indicates that asterisks need to be URL encoded to be valid, so shouldn't the Twitter gem be handling this for me?
https://twittercommunity.com/t/asterisk/6343

[geori]

I'm having the same issue when posting @ or # in my search terms. I can't figure out where the conflict is, so I had to revert to 5.13.0 for my app to run properly.

[iandawg]

I've found that the same results occur using the https://dev.twitter.com/rest/tools/console. This seems to be an issue with the API, not with the gem. Any thoughts?

[dredenba]

Is there a work around for this? I thought I would be able to URL encode it manually if this gem doesn't do it but I don't get * when I use %2A

[renatolond]

Sorry to re-raise this topic, but I'm getting the same on master. I am going around temporarily by replacing asterisk by ＊. I also tried what dredenba suggested of using the url encoded version and it does not work.

[alzeih]

I think I've figured out the cause of "Could not authenticate you" with *.

https://github.com/sferik/twitter/blob/master/lib/twitter/rest/request.rb#L28 contains a call to set_multipart_options!(request_method, options) which does

@headers = Twitter::Headers.new(@client, @request_method, @uri, options).request_headers

This creates the oauth http headers, which signs the contents of the request, including the options.

When the request is sent

response = http_client.headers(@headers).public_send(@request_method, @uri.to_s, options_key => @options)

the http library encodes the form data, which is fine in most cases, but with * (and likely some of the other specials in https://tools.ietf.org/html/rfc3986#section-2.2 the http library encodes the values, which changes the contents of the request.

Since the request is now different, the twitter api rejects the request with a 401 Unauthorized.

[alzeih]

Actually, after reading #888, it's the other way around, it's not being encoded.

It seems related to httprb/http#449 , which suggests a workaround.

This patch works for me.

diff --git a/lib/twitter/rest/request.rb b/lib/twitter/rest/request.rb
index db34abbe..1c12c18a 100644
--- a/lib/twitter/rest/request.rb
+++ b/lib/twitter/rest/request.rb
@@ -32,8 +32,8 @@ module Twitter

       # @return [Array, Hash]
       def perform
-        options_key = @request_method == :get ? :params : :form
-        response = http_client.headers(@headers).public_send(@request_method, @uri.to_s, options_key => @options)
+        @uri.query_values = @options
+        response = http_client.headers(@headers).public_send(@request_method, @uri.to_s)
         response_body = response.body.empty? ? '' : symbolize_keys!(response.parse)
         response_headers = response.headers
         fail_or_return_response_body(response.code, response_body, response_headers)

[CedricBm]

Interesting patch @alzeih. However, this passes the tweet body into the URL as a GET parameter. So there are chances that we receive a 414 URI too long error, especially when sending direct messages which can have up to 10 000 characters.

[alzeih]

@CedricBm it also doesn't work with image uploads - although not because of length, but due to how query_values works. This patch is just demonstrating why asterisks cause the "Could not authenticate you" error.

[christianstanfield]

I'm also running into the same issue and can confirm that URI#encode_www_form inside HTTP::FormData does not convert asterisks, whereas Twitter::Headers is relying on the gem simple_oauth which is using URI::Parser and is encoding asterisks: URI::Parser.new.escape('*', /[^a-z0-9\-\.\_\~]/i) => "%2A". (https://github.com/laserlemon/simple_oauth/blob/be8a895c73a433dce47aa6ca37dd03a0a343fd55/lib/simple_oauth/header.rb#L29) Looks very much like there's an encoding incompatibility between these two.

[FabienChaynes]

Hi,

The 2.1.1 version of http-form_data includes a fix for this case discussed in httprb/form_data#22 and implemented in httprb/form_data@3bf2e83.

It allows to override the http-form_data encoding method.

The following snippet overrides the encoding to act like the simple_oauth gem and thus fix the issue (you can put it in config/initializers/http_form_data.rb for example):

require "uri"

# The encoder method of the http gem needs to be overriden because of the twitter gem.
# Without that, there's an incompatibility between the simple_oauth gem which encodes asterisks and the http one which does not.
# Cf. https://github.com/httprb/form_data/issues/22 and https://github.com/sferik/twitter/issues/677
HTTP::FormData::Urlencoded.encoder = lambda do |enum|
  unescaped_chars = /[^a-z0-9\-\.\_\~]/i
  enum.map do |k, v|
    if v.nil?
      ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
    elsif v.respond_to?(:to_ary)
      v.to_ary.map do |w|
        str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
        unless w.nil?
          str << '='
          str << ::URI::DEFAULT_PARSER.escape(w.to_s, unescaped_chars)
        end
      end.join('&')
    else
      str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars)
      str << '='
      str << ::URI::DEFAULT_PARSER.escape(v.to_s, unescaped_chars)
    end
  end.join('&')
end

It's been a few weeks we're using this method and it seems fine so far. Be careful though, it can have some side effects if some of the other gems you're using depend on http-form_data and expect the original encoding way.

[kieraneglin]

Reviving this one to say it's still affecting the latest version