New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error "BUG: missing headers nil" #81

Open
fkhodkov opened this Issue Jan 11, 2019 · 59 comments

Comments

@fkhodkov
Copy link

fkhodkov commented Jan 11, 2019

I used magithub for some time, and today it stopped working. After some investigation I found that it seems to be ghub-related.

For example, I get an error after doing this:

  1. Remove access token in github settings
  2. Removing ^magithub line from .authinfo.gpg
  3. M-x auth-source-forget-all-cached
  4. M-x ghub-create-token
    After entering Host, Username, Package and not modifying Scopes, I get the following backtrace:
Debugger entered--Lisp error: (error "BUG: missing headers nil")
  signal(error ("BUG: missing headers nil"))
  error("BUG: missing headers %s" nil)
  ghub--handle-response-headers(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/authorizations" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x29d0ab5>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--handle-response(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/authorizations" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x29d0ab5>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--retrieve("{\"scopes\":[\"repo\",\"user\",\"notifications\"],\"note\":\"Emacs package magithub @ fkhodkov-note\"}" #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/authorizations" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x29d0ab5>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>)("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil)
  apply(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil))
  magithub-debug--ghub-request-wrapper(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) "POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil)
  apply(magithub-debug--ghub-request-wrapper #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil))
  ghub-request("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil)
  ghub-post("/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :username "fkhodkov" :auth basic :host "api.github.com")
  ghub-create-token("api.github.com" "fkhodkov" magithub ("repo" "user" "notifications"))
  funcall-interactively(ghub-create-token "api.github.com" "fkhodkov" magithub ("repo" "user" "notifications"))
  call-interactively(ghub-create-token record nil)
  command-execute(ghub-create-token record)
  execute-extended-command(nil "ghub-create-token" nil)
  funcall-interactively(execute-extended-command nil "ghub-create-token" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

, even though in github settings I see newly-created token. I can regenerate token and manually store it in .authinfo.gpg, but then I'll get similar error after every attempt to do anything magithub-related. For example, this I get after trying to magithub-clone the ghub's repo:

Debugger entered--Lisp error: (error "BUG: missing headers nil")
  signal(error ("BUG: missing headers nil"))
  error("BUG: missing headers %s" nil)
  ghub--handle-response-headers(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/repos/magit/ghub" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "GET" :headers #f(compiled-function () #<bytecode 0x273b74d>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--handle-response(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/repos/magit/ghub" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "GET" :headers #f(compiled-function () #<bytecode 0x273b74d>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--retrieve(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/repos/magit/ghub" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "GET" :headers #f(compiled-function () #<bytecode 0x273b74d>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>)("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil)
  apply(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil))
  magithub-debug--ghub-request-wrapper(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) "GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil)
  apply(magithub-debug--ghub-request-wrapper #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil))
  ghub-request("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil)
  ghubp-request(get "/repos/magit/ghub" nil nil)
  apply(ghubp-request get "/repos/magit/ghub" (nil nil))
  ghubp-get-repos-owner-repo(((owner (login . "magit")) (name . "ghub")))
  byte-code("\302 \303 C\304\030\211\305\306B\002\242B\240\210\307\310\311\312\313\005!\314\"\315$\020\316\002!)\262\001\206:\0\211\317\320\002\236A\236A\321\002\236A\322\323\003\003#\266\202\262\001\324\321\002\"\325\326\011\003\304\005%\002\001D\207" [ghubp-contextualize-function magithub-clone-default-directory magithub-clone--get-repo ghubp-get-context nil auth magithub make-byte-code 0 "\300\242\207" vconcat vector [] 1 ghubp-get-repos-owner-repo login owner name user-error "Repository %s/%s does not exist" alist-get read-directory-name "Destination: "] 9)
  call-interactively(magithub-clone record nil)
  command-execute(magithub-clone record)
  execute-extended-command(nil "magithub-clone" nil)
  funcall-interactively(execute-extended-command nil "magithub-clone" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)
@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Jan 11, 2019

I don't think it is a regression on our side. I haven't gotten this error in a while (though users with huge repositories have), but today I have a few times. I think this is a symptom of something having gone wrong on the server, but I might be wrong.

Edit: As noted below by @vermiculus another possibility is that we get a valid response but a but in url.el then causes it to be discarded.

@iocanel

This comment has been minimized.

Copy link

iocanel commented Jan 12, 2019

I am hitting the same issue, after upgrading to the latest version.

@pbiggar

This comment has been minimized.

Copy link

pbiggar commented Jan 13, 2019

I'm also hitting this issue, as of 3-4 days ago, when I use magithub and it attempts to get the github status header, issues section, or pull requests section.

@humitos

This comment has been minimized.

Copy link

humitos commented Jan 15, 2019

I'm hitting this issue also with magithub and with forge as well. I'm happy providing any help that I can to debug it. Let me know.

@d12frosted

This comment has been minimized.

Copy link

d12frosted commented Jan 15, 2019

Exactly the same issue happens to me. Any workarounds?

@d12frosted

This comment has been minimized.

Copy link

d12frosted commented Jan 15, 2019

I've tried to debug it, but I don't understand how url-http-end-of-headers is being set. I see it's defined here and then it's used here before the described error is fired.

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Jan 15, 2019

I've run into this problem before during development. Google (or whatever) search "url-http-end-of-headers" allred. I'm on mobile or I'd link things directly.

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Jan 15, 2019

https://emacs.stackexchange.com/a/32952/2264 I don't believe I ever actually submitted a bug report but I may just be forgetting. If I did, I don't think I heard anything back.

@humitos

This comment has been minimized.

Copy link

humitos commented Jan 15, 2019

@vermiculus thanks for the info! However, I'm not being able to write a workaround if there is any. Was anyone able to make it work? I'd appreciate if you share the solution here. Thanks!

@tarsius tarsius changed the title error "BUG: missing headers nil" after any github operation error "BUG: missing headers nil" Jan 18, 2019

@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Jan 18, 2019

Changed the title because this can also happen with Gitlab.

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Jan 18, 2019

@humitos I don't believe I was able to find a solution; just sharing my investigation :-)

@yssource

This comment has been minimized.

Copy link

yssource commented Jan 21, 2019

I've run into this problem also.

@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Jan 21, 2019

@vermiculus I would very much appreciate it if you could investigate this further.

@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Jan 21, 2019

I found a possible work-around:

diff --git a/ghub.el b/ghub.el
index 5a1cfc8..b7eef54 100644
--- a/ghub.el
+++ b/ghub.el
@@ -738,7 +738,8 @@ (defun ghub--basic-auth-errorback (url &optional prompt _overwrite _realm _args)
     (if (assoc "X-GitHub-OTP" (ghub--handle-response-headers nil nil))
         (progn
           (setq url-http-extra-headers
-                `(("Content-Type" . "application/json")
+                `(("Pragma" . "no-cache")
+                  ("Content-Type" . "application/json")
                   ("X-GitHub-OTP" . ,(ghub--read-2fa-code))
                   ;; Without "Content-Type" and "Authorization".
                   ;; The latter gets re-added from the return value.

Please try that (making sure to recompile ghub.el after making this change).

@krobelus

This comment was marked as off-topic.

Copy link

krobelus commented Jan 21, 2019

When I call (magit-refresh) it still fails in (ghubp-ratelimit) before calling the function where your fix is applied.
Looks like I have exceeded the rate limit. So this is another possible source of trouble. I'll try again later.

curl https://api.github.com/rate_limit -sH'Content-Type: application/json' | jq
{
  "resources": {
    "core": {
      "limit": 60,
      "remaining": 60,
      "reset": 1548103605
    },
    "search": {
      "limit": 10,
      "remaining": 10,
      "reset": 1548100065
    },
    "graphql": {
      "limit": 0,
      "remaining": 0,
      "reset": 1548103605
    }
  },
  "rate": {
    "limit": 60,
    "remaining": 60,
    "reset": 1548103605
  }
}
@tarsius

This comment was marked as off-topic.

Copy link
Member

tarsius commented Jan 21, 2019

There appears to be a regression on Github's side.

I get the exact same numbers (except for the reset date), but I do not get any errors.

Also graphql.limit is zero. That cannot be right as it would mean "you are not allowed to make any graphql requests whatsoever". Also the example output at https://developer.github.com/v3/rate_limit/ shows that the other limits should be two magnitudes bigger. "A few thousand" is also the number I remember from requests made a few months ago.

@fkhodkov

This comment has been minimized.

Copy link
Author

fkhodkov commented Jan 21, 2019

@tarsius still getting the same error after applying your fix.

@tarsius

This comment was marked as off-topic.

Copy link
Member

tarsius commented Jan 21, 2019

When I call (magit-refresh) it still fails in (ghubp-ratelimit) before calling the function where your fix is applied.

Ah, so you are using Magithub. @vermiculus It seems you have to remove the rate limiting code because Github returns invalid information.

@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Jan 21, 2019

@fkhodkov Lets find out if we actually get a 304. Please add a debug statement to url-http-parse-headers. Set url-debug to t and check if any Extracting document from cache... are inserted into the *URL-DEBUG* buffer.

@tarsius

This comment was marked as off-topic.

Copy link
Member

tarsius commented Jan 21, 2019

@krobelus What you have reported doesn't really seem related to the issue being discussed here. I am hidding the posts related to it to make this issue more readable. If you want to discuss this further please open a new issue.

@fkhodkov

This comment has been minimized.

Copy link
Author

fkhodkov commented Jan 21, 2019

No, no line with Extracting document from cache... in *URL-DEBUG*.

@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Jan 21, 2019

What did "Parsed HTTP headers: class=%d status=%d" produce?

@fkhodkov

This comment has been minimized.

Copy link
Author

fkhodkov commented Jan 21, 2019

Hmm, I don't see anything like this in *URL-DEBUG* buffer (I tried to run ghub-create-token -- should I run something else?).

@charignon

This comment has been minimized.

Copy link

charignon commented Jan 22, 2019

@tarsius, I made a couple of observations:

  • I can repro this behavior consistently when calling ghub directly (evaluating elisp) in synchronous mode. This issue does not repro when using the asynchronous mode (I tried over 50 runs).
  • If I put a breakpoint in url-retrieve-synchronously and call the magithub dashboard, it works consistently (where it fails consistently when not having the breakpoint to slow things down). From what I can tell, url-retrieve-synchronously has not changed recently. Maybe something changed on the github side that url-retrieve-synchronously does not know how to handle?

How else could we help debug that?

@brainlessdeveloper

This comment has been minimized.

Copy link

brainlessdeveloper commented Jan 24, 2019

I'm also having this problem using Magithub, I'll try to find some time this weekend to help debugging it.

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Jan 26, 2019

I'm also now receiving this error after updating. Here's the debug information it looks like you added:

error in process filter: ghub--handle-response-headers: BUG: missing headers
  See https://github.com/magit/ghub/issues/81.
  headers: 0
  status: nil
  buffer: #<buffer  *http api.github.com:443*-989552>
  buffer-string:
  "^M
"

Am I to understand that the buffer provided by url.el is effectively empty?

@hpdeifel

This comment has been minimized.

Copy link

hpdeifel commented Jan 26, 2019

FWIW, I can consistently make it work by doing (trace-function 'url-retrieve-synchronously)...

@hoarf

This comment has been minimized.

Copy link

hoarf commented Feb 3, 2019

Also happening to me for a while now.
GNU Emacs 26.1 with magithub and development branch of spacemacs at 990290e24
Arch Linux

@mfiano

This comment has been minimized.

Copy link

mfiano commented Feb 4, 2019

This seems to be an issue with the server response, as it occurs for me with both ghub.el and gh.el (which gist.el uses). I am currently unable to even submit a gist using a completely different GitHub API client.

Edit: This has been ongoing for over a month with both Emacs packages attempting to communicate with GitHub, and no solution in sight.

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Feb 4, 2019

Has anyone contacted GitHub about this issue yet? If we can find a curl command that can reproduce this issue, we can raise it to support.

@charignon

This comment has been minimized.

Copy link

charignon commented Feb 4, 2019

From looking at packet trace with Wireshark, it seems that the error is not originating from Github's side.

Here is what I observed: when we get blank headers in the synchronous case, the client (emacs) sends a TCP RST packet proactively right after making the HTTP request. In other words, emacs closes the connection with Github before getting the response. Therefore the response is parsed as blank and the headers are nil. This seems to indicate an issue with network process / connection management on the client side.

My best guess so far is that, the issue is a race in connection management at connection establishment-time, based on two observations:

  1. Adding sleeps before the point where we send the http request consistently fixes the issue for me
  2. The requests consistently work when a connection with github is already established as an emacs process (without modifying any code and inserting sleeps) that can be reused.

Of course, the race could have been made more likely by something that Github changed (timing perhaps), but I don't think that it is an issue with the validity of their response or their server not respecting the standard.

@vermiculus

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Feb 4, 2019

We're lucky you have those skills 😄 I'm lost with tools like Wireshark -- thanks for the info!

Do you think this is an issue that should be fixed in url.el? Is a temporary kludge in ghub--retrieve (i.e., that sleep()) appropriate for now?

@charignon

This comment has been minimized.

Copy link

charignon commented Feb 5, 2019

I did some more experimentation, tracing everything in url-http.el and url.el I wasn't able to spot the root cause.

In the spirit of finding a solution, I played around with request.el using this mimum example (replace the token):

(request
 "https://api.github.com/user/repos"
 :headers '(("Authorization" . "token XXX"))
 :parser 'json-read
 :sync t))

request.el lets you pick a backend: curl or url.el (the one that ghub uses). With the url.el one the snippet above fails (same symptoms as this discussion), with the curl one it works.

So, to fix this without doing a hack, let's change ghub to use request.el, it shouldn't be too much work because the code is pretty well factored already. I will take a look when I have time unless someone gets to it first (@tarsius FYI).

@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Feb 5, 2019

Thanks @charignon for looking into this!

The best course of action would be to talk to the Emacs maintainers about this.I think (but could be completely wrong) that the bug is at the C level. When I poked around the elisp code I too didn't find anything.

Could you please take care of opening an Emacs bug report with steps to reproduce and maybe the output of wireshark? (Use url-retrieve directly, no ghub wrappers. If possible use a resource that does not require a token.)

Using request.el should only be a last resort. In fact if we do give up on url.el and switch to curl, then I would prefer using a thin module around libcurl instead. Also I think that switching to request.el would actually be quite a bit of work.

@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Feb 5, 2019

Ps: I won't be around until the end of the week.

@charignon

This comment has been minimized.

Copy link

charignon commented Feb 6, 2019

I found a minimum repro case with emacs -Q and submitted a bug report. I am waiting for it to show up in the tracker. Here is a gist with the bug report if anyone is interested.

If we switch to using curl instead of url.el, ghub will need curl installed as a dependency. I think curl is installed by default on Mac OS and probably most Linux installation, but not on Windows. I don't know how much it matters (it all depends on the user base and goals of the project).

I will wait for more input on this before sending out a PR.

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Feb 6, 2019

It wouldn't be without precedent to require Windows users to have otherwise-standard software installed. This is the case with Forge's SQL backend, for instance. I can understand the hesitation to add another dependency indefinitely, but it may be good as a temporary measure until the core bug is addressed.

Out of curiosity, is it an existing option to install emacs with libcurl? On my phone or I'd check.

@charignon

This comment has been minimized.

Copy link

charignon commented Feb 6, 2019

Out of curiosity, I tried compiling the latest emacs to check if the issue still happens there and it does not! 😄

The issue is fixed for me on the latest master. emacs-mirror/emacs@e87e6a2 fixed it and was merged mid january (@eggert Thanks!!!)
So as long as you are on a more recent version than that, it should work.

Can anyone else verify this as well? @vermiculus perhaps?

tarsius added a commit that referenced this issue Feb 6, 2019

ghub--retrieve: For Github make sync requests async
Work around a GnuTLS bug, which has been fixed in Emacs 27 and which,
in our case, appears to only affect connections to Github.

Re #81.
@tarsius

This comment has been minimized.

Copy link
Member

tarsius commented Feb 6, 2019

Ps: I won't be around until the end of the week.

Yeah well, sleeping for 12 hours straight seems to have done the trick.

The issue is fixed for me on the latest master.

Great! (Thanks so much and I am glad I didn't try to tackle this before you investigated; I would probably have barked at the wrong tree. 😝 )

Of course we will have to wait a decade until everyone has switched to Emacs 27. Meanwhile we need some kludge.

Since this apparently only happens when using url-retrieve-synchronously we can just use url-retrieve and then block until that is done. I have pushed a commit implementing that approach to the 81-kludge branch. Please give that a try too.

@brainlessdeveloper

This comment has been minimized.

Copy link

brainlessdeveloper commented Feb 6, 2019

For any Arch users interested in (master branch) Emacs 27: https://aur.archlinux.org/packages/emacs-git

@vermiculus

This comment has been minimized.

Copy link
Contributor

vermiculus commented Feb 6, 2019

Can anyone else verify this as well? @vermiculus perhaps?

I've not been able to reproduce this issue for some time, but I'm sure others could corroborate your results.

@egh

This comment has been minimized.

Copy link

egh commented Feb 7, 2019

Is this the same issue as https://debbugs.gnu.org/cgi/bugreport.cgi?bug=14983 ?

@mfiano

This comment has been minimized.

Copy link

mfiano commented Feb 7, 2019

I can confirm that this issue is fixed on Emacs git master (and also the similar issue with gist.el I mentioned above)

@asheshambasta

This comment was marked as off-topic.

Copy link

asheshambasta commented Feb 8, 2019

@brainlessdeveloper how stable is Emacs 27 currently? I use emacs for my bread and butter so having it crash due to some other bug will not be ideal. :-)

@brainlessdeveloper

This comment was marked as off-topic.

Copy link

brainlessdeveloper commented Feb 8, 2019

@asheshambasta I use Emacs exclusively for Magit, inside of other editors. My view on how master branch Emacs 27 works right now is definitely skewed, and it might change tomorrow, but for now it works flawlessly for me.

@basil-conto

This comment was marked as off-topic.

Copy link
Contributor

basil-conto commented Feb 8, 2019

@asheshambasta I've been building Emacs master regularly and effectively living in it on a daily basis for the last couple of years. It is rare IME that I come across a problem which isn't also present in other releases. I've only had a single crash, for example, and that was also present in the Emacs 26 pretest and was quickly fixed. Having said that, I also follow the emacs-devel and bug-gnu-emacs mailing lists and read through commits before building, so I have a fair idea of when there's a higher chance of instability, e.g. when a major low-level feature like portable dumping is added. YMMV ;)

@asheshambasta

This comment was marked as off-topic.

Copy link

asheshambasta commented Feb 8, 2019

@basil-conto thanks for the heads up, I'm tempted to try it and I will over this weekend (or maybe even today, since I miss magit too much).

Edit: using emacs-git and things work normally.

@genovese

This comment has been minimized.

Copy link

genovese commented Feb 8, 2019

@genovese

This comment has been minimized.

Copy link

genovese commented Feb 8, 2019

@pbiggar

This comment has been minimized.

Copy link

pbiggar commented Feb 11, 2019

Following the suggestions in this thread, I fixed this on OSX by installing the latest emacs: brew install emacs --HEAD

@jperras

This comment has been minimized.

Copy link

jperras commented Feb 11, 2019

I upgraded to emacs 27 on macOS via brew install emacs --HEAD as indicated by @pbiggar, and the issue resolved itself for me as well.

@erkiesken

This comment has been minimized.

Copy link

erkiesken commented Feb 13, 2019

I am using emacs-plus package and also can confirm that after brew install emacs-plus --HEAD (now on Emacs 27.0.5) the missing headers error went away and my forge-pull finally succeeded.

@aaronjensen

This comment has been minimized.

Copy link

aaronjensen commented Feb 15, 2019

I still see this w/ a freshly built brew install emacs-plus --HEAD on macOS 10.14.2.

@aaronjensen

This comment has been minimized.

Copy link

aaronjensen commented Feb 15, 2019

For those of you who say it's working now, can you try M-x forge-reset-database and then M-x forge-pull? That triggers it for me sometimes. It is intermittent, however.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment