error "BUG: missing headers nil" #81

fkhodkov · 2019-01-11T16:23:24Z

TL;DR

Added by @tarsius.

We found and patched two bugs in Emacs that resulted in these symptoms.
See https://github.com/magit/ghub/wiki/Known-Issues#emacs-bugs-resulting-in-bug-missing-headers.

I used magithub for some time, and today it stopped working. After some investigation I found that it seems to be ghub-related.

For example, I get an error after doing this:

Remove access token in github settings
Removing ^magithub line from .authinfo.gpg
M-x auth-source-forget-all-cached
M-x ghub-create-token
After entering Host, Username, Package and not modifying Scopes, I get the following backtrace:

Debugger entered--Lisp error: (error "BUG: missing headers nil")
  signal(error ("BUG: missing headers nil"))
  error("BUG: missing headers %s" nil)
  ghub--handle-response-headers(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/authorizations" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x29d0ab5>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--handle-response(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/authorizations" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x29d0ab5>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--retrieve("{\"scopes\":[\"repo\",\"user\",\"notifications\"],\"note\":\"Emacs package magithub @ fkhodkov-note\"}" #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/authorizations" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "POST" :headers #f(compiled-function () #<bytecode 0x29d0ab5>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>)("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil)
  apply(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil))
  magithub-debug--ghub-request-wrapper(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) "POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil)
  apply(magithub-debug--ghub-request-wrapper #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil))
  ghub-request("POST" "/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username "fkhodkov" :auth basic :host "api.github.com" :callback nil :errorback nil :extra nil)
  ghub-post("/authorizations" ((scopes "repo" "user" "notifications") (note . "Emacs package magithub @ fkhodkov-note")) :username "fkhodkov" :auth basic :host "api.github.com")
  ghub-create-token("api.github.com" "fkhodkov" magithub ("repo" "user" "notifications"))
  funcall-interactively(ghub-create-token "api.github.com" "fkhodkov" magithub ("repo" "user" "notifications"))
  call-interactively(ghub-create-token record nil)
  command-execute(ghub-create-token record)
  execute-extended-command(nil "ghub-create-token" nil)
  funcall-interactively(execute-extended-command nil "ghub-create-token" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

, even though in github settings I see newly-created token. I can regenerate token and manually store it in .authinfo.gpg, but then I'll get similar error after every attempt to do anything magithub-related. For example, this I get after trying to magithub-clone the ghub's repo:

Debugger entered--Lisp error: (error "BUG: missing headers nil")
  signal(error ("BUG: missing headers nil"))
  error("BUG: missing headers %s" nil)
  ghub--handle-response-headers(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/repos/magit/ghub" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "GET" :headers #f(compiled-function () #<bytecode 0x273b74d>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--handle-response(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/repos/magit/ghub" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "GET" :headers #f(compiled-function () #<bytecode 0x273b74d>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  ghub--retrieve(nil #s(ghub--req :url #s(url :type "https" :user nil :password nil :host "api.github.com" :portspec nil :filename "/repos/magit/ghub" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous nil) :forge nil :silent nil :method "GET" :headers #f(compiled-function () #<bytecode 0x273b74d>) :handler ghub--handle-response :unpaginate nil :noerror nil :reader nil :callback nil :errorback nil :value nil :extra nil))
  #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>)("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil)
  apply(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil))
  magithub-debug--ghub-request-wrapper(#f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) "GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil)
  apply(magithub-debug--ghub-request-wrapper #f(compiled-function (arg1 arg2 &rest rest) "Make a request for RESOURCE and return the response body.\n\nAlso place the response headers in `ghub-response-headers'.\n\nMETHOD is the HTTP method, given as a string.\nRESOURCE is the resource to access, given as a string beginning\n  with a slash.\n\nPARAMS, QUERY, PAYLOAD and HEADERS are alists used to specify\n  data.  The Github API documentation is vague on how data has\n  to be transmitted and for a particular resource usually just\n  talks about \"parameters\".  Generally speaking when the METHOD\n  is \"HEAD\" or \"GET\", then they have to be transmitted as a\n  query, otherwise as a payload.\nUse PARAMS to automatically transmit like QUERY or PAYLOAD would\n  depending on METHOD.\nUse QUERY to explicitly transmit data as a query.\nUse PAYLOAD to explicitly transmit data as a payload.\n  Instead of an alist, PAYLOAD may also be a string, in which\n  case it gets encoded as UTF-8 but is otherwise transmitted as-is.\nUse HEADERS for those rare resources that require that the data\n  is transmitted as headers instead of as a query or payload.\n  When that is the case, then the API documentation usually\n  mentions it explicitly.\n\nIf SILENT is non-nil, then don't message progress reports and\n  the like.\n\nIf UNPAGINATE is t, then make as many requests as necessary to\n  get all values.  If UNPAGINATE is a natural number, then get\n  at most that many pages.  For any other non-nil value raise\n  an error.\nIf NOERROR is non-nil, then do not raise an error if the request\n  fails and return nil instead.  If NOERROR is `return', then\n  return the error payload instead of nil.\nIf READER is non-nil, then it is used to read and return from the\n  response buffer.  The default is `ghub--read-json-payload'.\n  For the very few resources that do not return JSON, you might\n  want to use `ghub--decode-payload'.\n\nIf USERNAME is non-nil, then make a request on behalf of that\n  user.  It is better to specify the user using the Git variable\n  `github.user' for \"api.github.com\", or `github.HOST.user' if\n  connecting to a Github Enterprise instance.\n\nEach package that uses `ghub' should use its own token. If AUTH\n  is nil, then the generic `ghub' token is used instead.  This\n  is only acceptable for personal utilities.  A packages that\n  is distributed to other users should always use this argument\n  to identify itself, using a symbol matching its name.\n\n  Package authors who find this inconvenient should write a\n  wrapper around this function and possibly for the\n  method-specific functions as well.\n\n  Some symbols have a special meaning.  `none' means to make an\n  unauthorized request.  `basic' means to make a password based\n  request.  If the value is a string, then it is assumed to be\n  a valid token.  `basic' and an explicit token string are only\n  intended for internal and debugging uses.\n\n  If AUTH is a package symbol, then the scopes are specified\n  using the variable `AUTH-github-token-scopes'.  It is an error\n  if that is not specified.  See `ghub-github-token-scopes' for\n  an example.\n\nIf HOST is non-nil, then connect to that Github instance.  This\n  defaults to \"api.github.com\".  When a repository is connected\n  to a Github Enterprise instance, then it is better to specify\n  that using the Git variable `github.host' instead of using this\n  argument.\n\nIf FORGE is `gitlab', then connect to Gitlab.com or, depending\n  on HOST, to another Gitlab instance.  This is only intended for\n  internal use.  Instead of using this argument you should use\n  function `glab-request' and other `glab-*' functions.\n\nIf CALLBACK and/or ERRORBACK is non-nil, then make one or more\n  asynchronous requests and call CALLBACK or ERRORBACK when\n  finished.  If no error occurred, then call CALLBACK, unless\n  that is nil.\n\n  If an error occurred, then call ERRORBACK, or if that is nil,\n  then CALLBACK.  ERRORBACK can also be t, in which case an error\n  is signaled instead.  NOERROR is ignored for all asynchronous\n  requests.\n\nBoth callbacks are called with four arguments.\n  1. For CALLBACK, the combined value of the retrieved pages.\n     For ERRORBACK, the error that occured when retrieving the\n     last page.\n  2. The headers of the last page as an alist.\n  3. Status information provided by `url-retrieve'. Its `:error'\n     property holds the same information as ERRORBACK's first\n     argument.\n  4. A `ghub--req' struct, which can be passed to `ghub-continue'\n     (which see) to retrieve the next page, if any." #<bytecode 0x15ef2e5>) ("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil))
  ghub-request("GET" "/repos/magit/ghub" nil :query nil :payload nil :unpaginate nil :headers nil :username nil :auth magithub :host nil)
  ghubp-request(get "/repos/magit/ghub" nil nil)
  apply(ghubp-request get "/repos/magit/ghub" (nil nil))
  ghubp-get-repos-owner-repo(((owner (login . "magit")) (name . "ghub")))
  byte-code("\302 \303 C\304\030\211\305\306B\002\242B\240\210\307\310\311\312\313\005!\314\"\315$\020\316\002!)\262\001\206:\0\211\317\320\002\236A\236A\321\002\236A\322\323\003\003#\266\202\262\001\324\321\002\"\325\326\011\003\304\005%\002\001D\207" [ghubp-contextualize-function magithub-clone-default-directory magithub-clone--get-repo ghubp-get-context nil auth magithub make-byte-code 0 "\300\242\207" vconcat vector [] 1 ghubp-get-repos-owner-repo login owner name user-error "Repository %s/%s does not exist" alist-get read-directory-name "Destination: "] 9)
  call-interactively(magithub-clone record nil)
  command-execute(magithub-clone record)
  execute-extended-command(nil "magithub-clone" nil)
  funcall-interactively(execute-extended-command nil "magithub-clone" nil)
  call-interactively(execute-extended-command nil nil)
  command-execute(execute-extended-command)

The text was updated successfully, but these errors were encountered:

tarsius · 2019-01-11T19:30:03Z

I don't think it is a regression on our side. I haven't gotten this error in a while (though users with huge repositories have), but today I have a few times. I think this is a symptom of something having gone wrong on the server, but I might be wrong.

Edit: As noted below by @vermiculus another possibility is that we get a valid response but a bug in url.el then causes it to be discarded.

iocanel · 2019-01-12T14:26:04Z

I am hitting the same issue, after upgrading to the latest version.

pbiggar · 2019-01-13T19:02:06Z

I'm also hitting this issue, as of 3-4 days ago, when I use magithub and it attempts to get the github status header, issues section, or pull requests section.

humitos · 2019-01-15T09:24:59Z

I'm hitting this issue also with magithub and with forge as well. I'm happy providing any help that I can to debug it. Let me know.

d12frosted · 2019-01-15T09:29:10Z

Exactly the same issue happens to me. Any workarounds?

d12frosted · 2019-01-15T09:38:42Z

I've tried to debug it, but I don't understand how url-http-end-of-headers is being set. I see it's defined here and then it's used here before the described error is fired.

vermiculus · 2019-01-15T14:38:41Z

I've run into this problem before during development. Google (or whatever) search "url-http-end-of-headers" allred. I'm on mobile or I'd link things directly.

vermiculus · 2019-01-15T14:40:10Z

https://emacs.stackexchange.com/a/32952/2264 I don't believe I ever actually submitted a bug report but I may just be forgetting. If I did, I don't think I heard anything back.

humitos · 2019-01-15T19:12:28Z

@vermiculus thanks for the info! However, I'm not being able to write a workaround if there is any. Was anyone able to make it work? I'd appreciate if you share the solution here. Thanks!

tarsius · 2019-01-18T02:28:33Z

Changed the title because this can also happen with Gitlab.

vermiculus · 2019-01-18T04:27:29Z

@humitos I don't believe I was able to find a solution; just sharing my investigation :-)

yssource · 2019-01-21T08:24:53Z

I've run into this problem also.

tarsius · 2019-01-21T15:10:53Z

@vermiculus I would very much appreciate it if you could investigate this further.

tarsius · 2019-01-21T15:32:15Z

I found a possible work-around:

diff --git a/ghub.el b/ghub.el
index 5a1cfc8..b7eef54 100644
--- a/ghub.el
+++ b/ghub.el
@@ -738,7 +738,8 @@ (defun ghub--basic-auth-errorback (url &optional prompt _overwrite _realm _args)
     (if (assoc "X-GitHub-OTP" (ghub--handle-response-headers nil nil))
         (progn
           (setq url-http-extra-headers
-                `(("Content-Type" . "application/json")
+                `(("Pragma" . "no-cache")
+                  ("Content-Type" . "application/json")
                   ("X-GitHub-OTP" . ,(ghub--read-2fa-code))
                   ;; Without "Content-Type" and "Authorization".
                   ;; The latter gets re-added from the return value.

Please try that (making sure to recompile ghub.el after making this change).

fkhodkov · 2019-01-21T20:16:46Z

@tarsius still getting the same error after applying your fix.

tarsius · 2019-01-21T20:25:03Z

@fkhodkov Lets find out if we actually get a 304. Please add a debug statement to url-http-parse-headers. Set url-debug to t and check if any Extracting document from cache... are inserted into the *URL-DEBUG* buffer.

fkhodkov · 2019-01-21T20:37:15Z

No, no line with Extracting document from cache... in *URL-DEBUG*.

tarsius · 2019-01-21T20:55:33Z

What did "Parsed HTTP headers: class=%d status=%d" produce?

fkhodkov · 2019-01-21T21:11:23Z

Hmm, I don't see anything like this in *URL-DEBUG* buffer (I tried to run ghub-create-token -- should I run something else?).

charignon · 2019-01-22T08:07:41Z

@tarsius, I made a couple of observations:

I can repro this behavior consistently when calling ghub directly (evaluating elisp) in synchronous mode. This issue does not repro when using the asynchronous mode (I tried over 50 runs).
If I put a breakpoint in url-retrieve-synchronously and call the magithub dashboard, it works consistently (where it fails consistently when not having the breakpoint to slow things down). From what I can tell, url-retrieve-synchronously has not changed recently. Maybe something changed on the github side that url-retrieve-synchronously does not know how to handle?

How else could we help debug that?

fnune · 2019-01-24T08:16:35Z

I'm also having this problem using Magithub, I'll try to find some time this weekend to help debugging it.

vermiculus · 2019-01-26T02:29:44Z

I'm also now receiving this error after updating. Here's the debug information it looks like you added:

error in process filter: ghub--handle-response-headers: BUG: missing headers
  See https://github.com/magit/ghub/issues/81.
  headers: 0
  status: nil
  buffer: #<buffer  *http api.github.com:443*-989552>
  buffer-string:
  "^M
"

Am I to understand that the buffer provided by url.el is effectively empty?

hpdeifel · 2019-01-26T07:26:49Z

FWIW, I can consistently make it work by doing (trace-function 'url-retrieve-synchronously)...

togakangaroo · 2021-03-12T05:35:56Z

Can this issue be reopened since its not resolved?

tarsius · 2021-03-12T09:34:00Z

If someone provides new information beyond "it happens for me too, sometimes", then I will investigate but since I think the issue is either on the server side or in url, I do not intend to reopen this issue until that happens. This issue has the faq label, which counts for something too.

ahonnecke · 2021-05-10T17:24:48Z

I'm seeing this in master

ahonnecke@pop-os:~/src/saris-tf$ emacs --version
GNU Emacs 28.0.50

error in process filter: ghub--handle-response-headers: BUG: missing headers
  See https://github.com/magit/ghub/issues/81.
  url: https://api.github.com/graphql
  headers: 0
  status: nil
  buffer: #<buffer  *http api.github.com:443*-196371>
error in process filter: BUG: missing headers
  See https://github.com/magit/ghub/issues/81.
  url: https://api.github.com/graphql
  headers: 0
  status: nil
  buffer: #<buffer  *http api.github.com:443*-196371>

Uninstalling forge, magit and ghub, then reinstalling same did not fix.

This fixed it:

(setq ghub-use-workaround-for-emacs-bug 'force)

kiniry · 2021-06-16T22:45:52Z

I'm seeing this with the following config:

macOS 10.15.7
Emacs 27.2
magit 20210616.1107
ghub 20210615.1504 (auto-installed by clean-installing magit)
forge 20210615.1040 (auto-installed by clean-installing magit)

The failure is triggered by innocuous projects' issues on a private Gitlab enterprise server (v13.12.2) or on GitHub.

anquegi · 2021-06-29T18:19:19Z

Same reproduced here with doom emacs native compilation enabled and:
Magit b68a760, Git 2.31.1, Emacs 28.0.50, darwin

error in process filter: ghub--handle-response-headers: BUG: missing headers
See #81.
url: https://api.github.com/graphql
headers: 0
status: nil
buffer: #<buffer http api.github.com:443-373895>
error in process filter: BUG: missing headers
See #81.
url: https://api.github.com/graphql
headers: 0
status: nil
buffer: #<buffer http api.github.com:443-373895>

But only for one repository the other work well

yosevu · 2021-08-04T12:36:41Z

(setq ghub-use-workaround-for-emacs-bug 'force) worked for me!

macOS Big Sur
Emacs 27.2

micah · 2022-04-08T01:58:59Z

I'm on Emacs 27.1 on Debian Linux (Bullseye), and I'm getting this error. I am using the latest ghub (20220403.1248) and magit (20220406.1950) and forge (20220407.1932). I initially tried to uninstall ghub, forge and magit, exit emacs and then start emacs again, but that did not solve the problem.

This issue is definitely not fixed in version 26.3 of Emacs.

I'm not on macOS.

In order for me to test this, I do M-x forge-reset-database, and then M-x forge-pull. I then have to wait a long time, because there are almost 2000 issues. The problem occurs at the same issue every single time (issue #1789).

I attempted to add (setq ghub-use-workaround-for-emacs-bug 'force) and re-did everything, and.... emacs was killed by the kernel OOM:

[32967.504131] Out of memory: Killed process 302561 (emacs) total-vm:13375876kB, anon-rss:12952964kB, file-rss:0kB, shmem-rss:28kB, UID:1000 pgtables:25600kB oom_score_adj:0

tarsius · 2022-04-08T12:03:45Z

I'm afraid it's still a mystery bug that I don't know how to fix.

micah · 2022-04-08T13:47:17Z

I upgraded to emacs 28.1, same problem. When I added the ghub-use-workaround-for-emacs-bug I just got the following in the minibuffer:

ghub--handle-response-headers: BUG: missing headers
  See https://github.com/magit/ghub/issues/81.
  url: https://code.mygitlab.org/api/v4/projects/34/issues/11677/notes?per_page=100
  headers: 0
  status: nil
  buffer: #<buffer  *http code.mygitlab.org:443*>

Is there a way that ghub can just skip an issue when ghub--handle-response-headers hits a missing header? Its this one issue that prevents me from using forge, an issue that I'm never going to need to even access via forge, so it being missing would not be a problem for me.

Interestingly, the one issue that this was getting stuck on was an issue that was imported from redmine to gitlab, and it had in its body only this:

<!-- @KB:{"andon":"none"} -->

I believe this was a tag added by a kanban add-on, at one point. I replaced that with some dummy text, and re-ran forge-pull and it then got stopped by a different issue, which also had this tag. I've now removed three of these and each time it stops somewhere else with that same tag. I'm unsure how many issues I have with this tag.

tarsius · 2022-04-08T14:50:50Z

Please keep such an issue so that we can use if for debugging purposes.

micah · 2022-04-08T14:53:51Z

I also have these in the body of issues that were imported from redmine:

*(from redmine: created on 2011-07-11)*

Looking at (defun ghub--handle-response-headers (status req) I see this:

    (while (re-search-forward "^\\([^:]*\\): \\(.+\\)"

micah · 2022-04-09T13:44:19Z

Unfortuantely, if I create an empty project, and add two issues, one with  and one with *(from redmine: created on 2011-07-11)* and then issue forge-pull on that project, it is able to successfully pull

micah · 2022-04-09T15:03:29Z

There were over 200 of these issues, I went in and removed these tags from all of them, and then forge-pull works.

tarsius · 2022-04-09T15:16:14Z

There were over 200 of these issues, I went in and removed these tags from all of them, and then forge-pull works.

That's too bad. Now I won't be able to debug it:

Please keep such an issue so that we can use if for debugging purposes.

micah · 2022-04-09T20:06:15Z

Unfortunately, there is something else going on, because I added those tags back, dropped the forge database, and can still do a forge-pull

nbarrientos · 2022-04-14T09:33:13Z

I'm hitting this as well rather often when downloading pullreqs from a Gitlab instance.

"GNU Emacs 28.0.92 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.33, cairo version 1.17.6)
 of 2022-03-29"

ghub                           20220403.1248  dependency            Client libraries for Git forge APIs.
forge                          20220407.1932  installed             Access Git forges from Magit.

error in process filter: ghub--handle-response-headers: BUG: missing headers
  See https://github.com/magit/ghub/issues/81.
  url: https://gitlab.cern.ch/api/v4/projects/2861
  headers: 0
  status: nil
  buffer: #<buffer  *http gitlab.cern.ch:443*>
error in process filter: BUG: missing headers
  See https://github.com/magit/ghub/issues/81.
  url: https://gitlab.cern.ch/api/v4/projects/2861
  headers: 0
  status: nil
  buffer: #<buffer  *http gitlab.cern.ch:443*>

Forge is able to download some pullreqs and the process stops mid-flight:

With this repo it stopped after processing 33 pullreqs.

Looking at *URL-DEBUG* here are the last 10 requests that the lib did:

(0) GET /api/v4/projects/2861 HTTP/1.1
(1) GET /api/v4/projects/2861 HTTP/1.1
(2) GET /api/v4/projects/2861/merge_requests/147/notes?per_page=100 HTTP/1.1
(3) GET /api/v4/projects/2861 HTTP/1.1
(4) GET /api/v4/projects/2861 HTTP/1.1
(5) GET /api/v4/projects/2861/merge_requests/146/notes?per_page=100 HTTP/1.1
(6) GET /api/v4/projects/2861 HTTP/1.1
(7) GET /api/v4/projects/2861 HTTP/1.1
(8) GET /api/v4/projects/2861/merge_requests/145/notes?per_page=100 HTTP/1.1
(9) GET /api/v4/projects/2861 HTTP/1.1

(generated by grepping for ^GET, hope they're in order)

I'm surprised it crashes when processing (9) because it managed to do that successfully several times. Maybe the process is async and the actual URL that's broken is (8)? When I manually cURL (5) and (8) the HTTP headers and the JSON look okay.

ghub-use-workaround-for-emacs-bug makes no difference, btw.

Let me know what else I can provide to help you debugging.

nbarrientos · 2022-04-14T10:15:42Z

I've added some debug code myself to ghub--handle-response-headers to print the size of the buffer containing the response for each URL, something like:

(defun ghub--handle-response-headers (status req)
  (goto-char (point-min))
  (forward-line 1)
  (message "%s: %d" (url-recreate-url (ghub--req-url req)) (buffer-size))
  (let (headers)
    (when (memq url-http-end-of-headers '(nil 0))
...

and indeed it's (9) above the problematic request:

https://gitlab.cern.ch/api/v4/projects/2861/merge_requests/146/notes?per_page=100: 5080
https://gitlab.cern.ch/api/v4/projects/2861: 4497 [2 times]
https://gitlab.cern.ch/api/v4/projects/2861/merge_requests/145/notes?per_page=100: 4067
https://gitlab.cern.ch/api/v4/projects/2861: 2

which is delivered for processing only with 2 chars. I doubt the response comes corrupted from the server so something must be broken down the line in ghub.el or url.el.

nbarrientos · 2022-04-16T16:19:44Z

I think I found the root cause, at least of my problem.

Emacs' url-http.el does not properly wait for the last CRLF when processing responses with chunked Transfer-Encoding (which Gitlab uses for some types of responses). If you look at url-http-wait-for-headers-change-function, you'll see that, when processing the last chunk of 0 bytes, the user callback is invoked no matter if the last \r\n has arrived or not:

(defun url-http-chunked-encoding-after-change-function (st nd length)
...
	    (if (= 0 url-http-chunked-length)
		(progn
		  ;; Found the end of the document!  Wheee!
		  (url-http-debug "Saw end of stream chunk!")
		  (setq read-next-chunk nil)
		  (url-display-percentage nil nil)
		  ;; Every chunk, even the last 0-length one, is
		  ;; terminated by CRLF.  Skip it.
		  (when (looking-at "\r?\n")
		    (url-http-debug "Removing terminator of last chunk")
		    (delete-region (match-beginning 0) (match-end 0)))
		  (if (re-search-forward "^\r?\n" nil t)
		      (url-http-debug "Saw end of trailers..."))
		  (if (url-http-parse-headers)
		      (url-http-activate-callback))))))))))

The problem is that if the last CRLF has not yet arrived the data is left in the socket and if the connection is reused (and it's likely to be reused), the first thing that url.el is going to see for the next request as response is \r\n, which will be processed by url-http-wait-for-headers-change-function as a "headerless malformed response" returning the control to the caller in a buffer with only two bytes (\r\n), no headers, etc.

This is rather difficult to trigger as "normally" the whole response has already arrived to the client when url-http-chunked-encoding-after-change-function is called and hence the last CRLF is already in the buffer and hence consumed. This patch fixes the issue for me:

@@ -36,6 +36,7 @@
 (defvar url-current-object)
 (defvar url-http-after-change-function)
 (defvar url-http-chunked-counter)
+(defvar url-http-chunked-last-crlf-missing nil)
 (defvar url-http-chunked-length)
 (defvar url-http-chunked-start)
 (defvar url-http-connection-opened)
@@ -1068,7 +1069,15 @@ the callback to be triggered."
 Cannot give a sophisticated percentage, but we need a different
 function to look for the special 0-length chunk that signifies
 the end of the document."
-  (save-excursion
+  (if url-http-chunked-last-crlf-missing
+      (progn
+        (goto-char url-http-chunked-last-crlf-missing)
+        (when (looking-at "\r\n")
+          (url-http-debug "Saw the last CRLF.")
+          (delete-region (match-beginning 0) (match-end 0))
+          (if (url-http-parse-headers)
+	         (url-http-activate-callback))))
+    (save-excursion
     (goto-char st)
     (let ((read-next-chunk t)
 	  (case-fold-search t)
@@ -1145,13 +1154,14 @@ the end of the document."
 		  (url-display-percentage nil nil)
 		  ;; Every chunk, even the last 0-length one, is
 		  ;; terminated by CRLF.  Skip it.
-		  (when (looking-at "\r?\n")
+		  (if (not (looking-at "\r?\n"))
+                      (setq-local url-http-chunked-last-crlf-missing (point))
 		    (url-http-debug "Removing terminator of last chunk")
-		    (delete-region (match-beginning 0) (match-end 0)))
-		  (if (re-search-forward "^\r?\n" nil t)
-		      (url-http-debug "Saw end of trailers..."))
-		  (if (url-http-parse-headers)
-		      (url-http-activate-callback))))))))))
+		    (delete-region (match-beginning 0) (match-end 0))
+		    (if (re-search-forward "^\r?\n" nil t)
+		        (url-http-debug "Saw end of trailers..."))
+		    (if (url-http-parse-headers)
+		        (url-http-activate-callback))))))))))))
 
 (defun url-http-wait-for-headers-change-function (_st nd _length)
   ;; This will wait for the headers to arrive and then splice in the

In the example list of requests of my post above, the request that's actually breaking things is (8) :)

I can consistently trigger the bug w/o the patch and once url-http.el is patched everything works as expected.

Probably worth a bug report for Emacs.

As per [0], the last chunk of 0 bytes is always accompanied by a last CRLF that signals the end of the message: chunked-body = *chunk last-chunk trailer-part CRLF ^ this one chunk = chunk-size [ chunk-ext ] CRLF chunk-data CRLF chunk-size = 1*HEXDIG last-chunk = 1*("0") [ chunk-ext ] CRLF chunk-data = 1*OCTET ; a sequence of chunk-size octets `url-http-chunked-encoding-after-change-function' is able to process (and remove) that terminator IF AVAILABLE in the buffer when processing the response, however it won't wait for it if it's not yet there. In other words: | Bottom of the response buffer | Bottom of the full response | | (visible to url-http) | (to be delivered to Emacs) | | ------------------------------+-----------------------------| | 0\r\n | 0\r\n | | | \r\n | If the last chunk is processed when the bottom of the response buffer is as above (note that the whole response has not yet been delivered to Emacs), url-http will call the user callback without waiting for the final terminator to be read from the socket. This is normally not an issue when doing one-shot requests, but it's problematic when the connection is reused immediately. As there are 2 bytes from the request N that have not been dealt with, they'll be considered as part of the response of the request N+1. On top, it turns out that when processing the headers of request N+1, `url-http-wait-for-headers-change-function' will consider the request a "headerless malformed response" delivering it broken to the caller. The proposed fix implements a state in which `url-http-chunked-encoding-after-change-function` properly waits for the very last element of the message preventing the problem explained above from happening. For additional context, this bug was found when debugging magit/ghub (see [1] for details). [0] https://datatracker.ietf.org/doc/html/rfc7230#section-4.1 [1] magit/ghub#81

nbarrientos · 2022-04-17T11:41:04Z

Probably worth a bug report for Emacs.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54989

As per [0], the last chunk of 0 bytes is always accompanied by a last CRLF that signals the end of the message: chunked-body = *chunk last-chunk trailer-part CRLF ^ this one chunk = chunk-size [ chunk-ext ] CRLF chunk-data CRLF chunk-size = 1*HEXDIG last-chunk = 1*("0") [ chunk-ext ] CRLF chunk-data = 1*OCTET ; a sequence of chunk-size octets `url-http-chunked-encoding-after-change-function' is able to process (and remove) that terminator IF AVAILABLE in the buffer when processing the response, however it won't wait for it if it's not yet there. In other words: | Bottom of the response buffer | Bottom of the full response | | (visible to url-http) | (to be delivered to Emacs) | | ------------------------------+-----------------------------| | 0\r\n | 0\r\n | | | \r\n | If the last chunk is processed when the bottom of the response buffer is as above (note that the whole response has not yet been delivered to Emacs), url-http will call the user callback without waiting for the final terminator to be read from the socket. This is normally not an issue when doing one-shot requests, but it's problematic when the connection is reused immediately. As there are 2 bytes from the request N that have not been dealt with, they'll be considered as part of the response of the request N+1. On top, it turns out that when processing the headers of request N+1, `url-http-wait-for-headers-change-function' will consider the request a "headerless malformed response" delivering it broken to the caller. The proposed fix implements a state in which `url-http-chunked-encoding-after-change-function` properly waits for the very last element of the message preventing the problem explained above from happening. For additional context, this bug was found when debugging magit/ghub (see [1] for details). [0] https://datatracker.ietf.org/doc/html/rfc7230#section-4.1 [1] magit/ghub#81

nbarrientos · 2022-04-19T07:26:09Z

The patch has been merged into Emacs' master and it should be already available in 29's snapshots. If you know a release manager to bribe so the patch is backported to 28.x it'll surely be money well spent (I've tried already).

In the meantime, users of Emacs 28 or older can always monkey patch the offending code in their configuration or build Emacs themselves.

Maybe it's worth patching ghub.el so when the bug is triggered in unpatched versions of Emacs a clearer hint about what versions of Emacs contain the fix is printed to the user (instead of hinting them to read this lengthy thread). Happy to send a patch over. Your call, @tarsius.

tarsius · 2022-04-19T17:45:17Z

Thanks so much for this, @nbarrientos!

I have asked in the debbugs thread for the fix to be included in 28.2.

I am also going to change the error message and provide instructions on how to use the fixed version of the faulty function. We got to do this anyway, even if it ends up being included in 28.2.

nbarrientos · 2022-04-21T06:59:06Z

I have asked in the debbugs thread for the fix to be included in 28.2.

Cool, although I might be looking in the wrong place but I can't see your message :/

I am also going to change the error message and provide instructions on how to use the fixed version of the faulty function. We got to do this anyway, even if it ends up being included in 28.2.

Nice, thanks.

tarsius · 2022-04-23T21:27:37Z

I have asked in the debbugs thread for the fix to be included in 28.2.

Cool, although I might be looking in the wrong place but I can't see your message :/

Hm, I cannot find it locally either. But I now think I won't push for the fix to be included in 28.2. We have to add a monkey patch anyway, for existing releases. And I don't feel like arguing against the policy of only fixing regressions. IMO it would make sense to make an exception here, but everyone feels like that about the bugs that affect them.

I am also going to change the error message and provide instructions on how to use the fixed version of the faulty function. We got to do this anyway, even if it ends up being included in 28.2.

Nice, thanks.

I have done that in 5eed205. I've already pushed that to master, but please have a look and let me know if anything seems off.

As 26faa2b943675107e1664b2fea7174137c473475 is not included I believe that the variable has to be buffer-local'ed before assignment. Relates to magit#81

nbarrientos · 2022-04-24T09:26:28Z

And I don't feel like arguing against the policy of only fixing regressions. IMO it would make sense to make an exception here, but everyone feels like that about the bugs that affect them

Yep, I tend to concur 😉

I have done that in 5eed205. I've already pushed that to master, but please have a look and let me know if anything seems off.

I believe there's something missing, I've sent a pull request. I haven't tried the hack itself, I was gonna but I saw the missing setq-local and I stopped as without it I don't think it'll work.

nbarrientos · 2022-04-24T13:27:32Z

Thanks again, @tarsius. The overridden function seems to work (Emacs 28.1). My reproducer is unable to trigger the bug with ghub 20220424.947 (without having to patch anything manually myself)

However, even though I'm very excited to see this sorted out, I see this approach as a dangerous bet (especially if enabled by default if (< emacs-major-version 29)) as, even though url-http.el is not changed very frequently:

The replaced function is not guaranteed to work with an older Emacs. It works with 28 but for instance in 27 it might be calling functions that are not declared in 27's url-http.el. Risk of meltdown 😉
Not sure having an Emacs package silently replacing core Emacs functionality by default is what users expect.

Indeed not having the fix in 28.x is a bummer but to be honest dunno if This Is The Way.

Of course you have the last word as maintainer, just my 2cts.

tarsius · 2022-04-24T17:13:25Z

The replaced function is not guaranteed to work with an older Emacs. It works with 28 but for instance in 27 it might be calling functions that are not declared in 27's url-http.el. Risk of meltdown

I had looked at all the commits that changed this function since 25.1. One fixes indentation, another removes some XEmacs-only text-properties, and 4f1df40db36b221e7842bd75d6281922dcb268ee seems to try to fix the same bug as your change but without fully succeeding (or maybe it's just a related bug). Regardless, assuming that including your bugfix is the right thing to do, then including this is also appropriate. I have extended the comment before the advise to add a note about this.

Not sure having an Emacs package silently replacing core Emacs functionality by default is what users expect.

It's probably not the Right Thing to Do™ but I am getting really sick of these url.el bugs and just want to be done with it. I did considered not enabling the advise by default, but decided to do it at least initially, and then wait to see if any users report issues that could be related to it. If not, then I intend to stick to it. Currently we know of at least four url (or url-related Emacs) bugs that affect Ghub; two of them are patched unconditionally and two are patched by default but with an option to disable the patch. Sadly, using url for anything that isn't so simple that you might as well just repeat the request if something doesn't work, requires patching its bugs.

tarsius changed the title ~~error "BUG: missing headers nil" after any github operation~~ error "BUG: missing headers nil" Jan 18, 2019

tarsius mentioned this issue Jan 18, 2019

Invalid search bound while fetching a gitlab merge request magit/forge#28

Closed

This comment has been minimized.

Sign in to view

vermiculus mentioned this issue Jan 22, 2019

Discussion from #81 #84

Closed

nbarrientos added a commit to nbarrientos/ghub that referenced this issue Apr 24, 2022

Make variable buffer local

5d8430f

As 26faa2b943675107e1664b2fea7174137c473475 is not included I believe that the variable has to be buffer-local'ed before assignment. Relates to magit#81

nbarrientos mentioned this issue Apr 24, 2022

Make variable buffer local #154

Closed

error "BUG: missing headers nil" #81

error "BUG: missing headers nil" #81

Comments

fkhodkov commented Jan 11, 2019 • edited by tarsius Loading

TL;DR

tarsius commented Jan 11, 2019 • edited Loading

iocanel commented Jan 12, 2019

pbiggar commented Jan 13, 2019 • edited Loading

humitos commented Jan 15, 2019

d12frosted commented Jan 15, 2019

d12frosted commented Jan 15, 2019 • edited Loading

vermiculus commented Jan 15, 2019

vermiculus commented Jan 15, 2019

humitos commented Jan 15, 2019

tarsius commented Jan 18, 2019

vermiculus commented Jan 18, 2019

yssource commented Jan 21, 2019

tarsius commented Jan 21, 2019

tarsius commented Jan 21, 2019

This comment has been minimized.

This comment has been minimized.

fkhodkov commented Jan 21, 2019

This comment has been minimized.

tarsius commented Jan 21, 2019

This comment has been minimized.

fkhodkov commented Jan 21, 2019

tarsius commented Jan 21, 2019

fkhodkov commented Jan 21, 2019

charignon commented Jan 22, 2019 • edited Loading

fnune commented Jan 24, 2019

vermiculus commented Jan 26, 2019

hpdeifel commented Jan 26, 2019

togakangaroo commented Mar 12, 2021

tarsius commented Mar 12, 2021

ahonnecke commented May 10, 2021

kiniry commented Jun 16, 2021

anquegi commented Jun 29, 2021 • edited Loading

yosevu commented Aug 4, 2021

micah commented Apr 8, 2022

tarsius commented Apr 8, 2022

micah commented Apr 8, 2022

tarsius commented Apr 8, 2022

micah commented Apr 8, 2022

micah commented Apr 9, 2022

micah commented Apr 9, 2022

tarsius commented Apr 9, 2022

micah commented Apr 9, 2022

nbarrientos commented Apr 14, 2022 • edited Loading

nbarrientos commented Apr 14, 2022

nbarrientos commented Apr 16, 2022 • edited Loading

nbarrientos commented Apr 17, 2022

nbarrientos commented Apr 19, 2022

tarsius commented Apr 19, 2022

nbarrientos commented Apr 21, 2022 • edited Loading

tarsius commented Apr 23, 2022

nbarrientos commented Apr 24, 2022

nbarrientos commented Apr 24, 2022

tarsius commented Apr 24, 2022

fkhodkov commented Jan 11, 2019 •

edited by tarsius

Loading

tarsius commented Jan 11, 2019 •

edited

Loading

pbiggar commented Jan 13, 2019 •

edited

Loading

d12frosted commented Jan 15, 2019 •

edited

Loading

charignon commented Jan 22, 2019 •

edited

Loading

anquegi commented Jun 29, 2021 •

edited

Loading

nbarrientos commented Apr 14, 2022 •

edited

Loading

nbarrientos commented Apr 16, 2022 •

edited

Loading

nbarrientos commented Apr 21, 2022 •

edited

Loading