Gitfs - No Content-Type header in response #38630

timwsuqld · 2017-01-09T05:05:21Z

Description of Issue/Question

Recently upgraded to 2016.11.1. We use gitfs_remotes to pull in a number of formulas. Started getting a lot of errors pulling gitfs remotes from Github. Our local Gitlab ones don't give any errors, but the local ones use git/ssh not https.

2017-01-09 14:56:30,499 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/chrony-formula.git': No Content-Type header in response
2017-01-09 14:56:31,455 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/epel-formula.git': No Content-Type header in response
2017-01-09 14:56:32,457 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/sudoers-formula.git': No Content-Type header in response
2017-01-09 14:56:33,434 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/firewalld-formula.git': No Content-Type header in response
2017-01-09 14:56:34,396 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/salt-formula.git': No Content-Type header in response
2017-01-09 14:56:35,352 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/postfix-formula.git': No Content-Type header in response
2017-01-09 14:56:36,327 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/saltstack-mariadb-formula.git': No Content-Type header in response
2017-01-09 14:56:37,290 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/fail2ban-formula.git': No Content-Type header in response
2017-01-09 14:56:38,263 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/aptly-formula.git': No Content-Type header in response
2017-01-09 14:56:39,263 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/iptables-formula.git': No Content-Type header in response
2017-01-09 14:56:40,238 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/ceph-formula.git': No Content-Type header in response
2017-01-09 14:57:43,987 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/saltstack-formulas/remi-formula.git': No Content-Type header in response
2017-01-09 14:57:45,073 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/zabbix-formula.git': No Content-Type header in response
2017-01-09 14:57:46,157 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/rsyslog-formula.git': No Content-Type header in response
2017-01-09 14:57:47,244 [salt.utils.gitfs ][ERROR   ][4841] Error occurred fetching gitfs remote 'https://github.com/suqld/etckeeper-formula.git': No Content-Type header in response

Setup

gitfs_remotes:
    - git@suq-version-control.suqld.org.au:it-internal/saltstack-config.git:
      - root: states
      - pubkey: /root/.ssh/id_rsa.pub
      - privkey: /root/.ssh/id_rsa
    - https://github.com/saltstack-formulas/remi-formula.git
    - https://github.com/suqld/zabbix-formula.git
    - https://github.com/suqld/rsyslog-formula.git
    - https://github.com/suqld/etckeeper-formula.git
    - https://github.com/suqld/ntp-formula.git
    - https://github.com/suqld/chrony-formula.git
    - https://github.com/suqld/epel-formula.git
    - https://github.com/suqld/sudoers-formula.git
    - https://github.com/suqld/firewalld-formula.git
    - https://github.com/suqld/salt-formula.git
    - https://github.com/suqld/postfix-formula.git
    - https://github.com/suqld/saltstack-mariadb-formula.git
    - https://github.com/suqld/fail2ban-formula.git
    - https://github.com/suqld/aptly-formula.git
    - https://github.com/suqld/iptables-formula.git
    - https://github.com/suqld/ceph-formula.git

Steps to Reproduce Issue

Lines when running log level debug

2017-01-09 15:02:49,054 [salt.utils.gitfs ][DEBUG   ][14968] Set update lock for gitfs remote 'https://github.com/suqld/chrony-formula.git'
2017-01-09 15:02:49,054 [salt.utils.gitfs ][DEBUG   ][14968] Fetching gitfs remote 'https://github.com/suqld/chrony-formula.git'
2017-01-09 15:02:50,125 [salt.utils.gitfs ][ERROR   ][14968] Error occurred fetching gitfs remote 'https://github.com/suqld/chrony-formula.git': No Content-Type header in response
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/gitfs.py", line 1404, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/usr/lib64/python2.7/site-packages/pygit2/remote.py", line 238, in fetch
    check_error(err)
  File "/usr/lib64/python2.7/site-packages/pygit2/errors.py", line 56, in check_error
    raise GitError(message)
GitError: No Content-Type header in response
2017-01-09 15:02:50,125 [salt.utils.gitfs ][DEBUG   ][14968] Removed update lock for gitfs remote 'https://github.com/suqld/chrony-formula.git'

Versions Report

Salt Version:
           Salt: 2016.11.1
 
Dependency Versions:
           cffi: 1.6.0
       cherrypy: 3.2.2
       dateutil: 2.5.3
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: 0.21.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.4
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-514.2.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

The text was updated successfully, but these errors were encountered:

gtmanfred · 2017-01-09T18:21:10Z

@terminalmage have you seen this error before with gitfs + pygit2?

Thanks,
Daniel

retrry · 2017-01-11T09:44:03Z

I don't remember seeing this error before. I have same problem, with same configuration

Versions Report

$ sudo salt --versions-report
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_checkout_notify_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_merge_tree_flag_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
Salt Version:
           Salt: 2016.11.1
 
Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.21.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.4
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-514.2.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

afletch · 2017-01-11T15:56:49Z

Also seeing this exact same issue with https remotes

# salt --versions-report
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_checkout_notify_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_merge_tree_flag_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 3.2.2
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.21.0
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.4
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-229.4.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

terminalmage · 2017-01-23T05:27:35Z

I've never seen this before. Those of you seeing this issue, are your masters behind a web proxy?

One common thread between all of you is that you are all on pygit2 0.21.4. The pygit2 project is known for making non-compatible changes in point releases (they even have a warning about this in their docs). However, 0.21.4 is not exactly a new release, so I have my doubts as to this being a sudden change in their API. I would have expected to have seen something like this by now if that were the case.

One other common thread in all three of you is that you're all using CentOS 7.3 with python-cffi 1.6.0. I know for a fact that there is currently a bug report filed against pygit2 in the RHEL/CentOS repositories, seemingly caused by the newer python-cffi. It does seem like there is an incompatibility with pygit2 and newer python-cffi, however whether that is fixed in newer pygit2 I am not sure. At the very least, there seems to be an incompatibility between pygit2 0.21.4 and python-cffi 1.6.0.

One thing you can try is to downgrade python-cffi to 0.8.6. Links to the RPMs are still available in the Fedora Buildsystem here.

afletch · 2017-01-23T09:39:52Z

If it helps, I resolved this by upgrading libgit to 0.25.1 and then installing the latest pygit2 using pip.

terminalmage · 2017-01-23T22:24:01Z

Yeah, it definitely appears to be an upstream issue.

terminalmage · 2017-01-24T16:09:30Z

I can confirm that I am now seeing this in cent7. Unfortunately there is nothing I can really do until RedHat fixes this issue on their end. If you don't need authenticated repos, maybe try installing GitPython and set gitfs_provider: gitpython in the master config file.

afletch · 2017-01-25T10:31:36Z

FWIW, my fix on CentOS7 went something like:

yum remove libgit2 python-pygit2
wget https://github.com/libgit2/libgit2/archive/v0.25.1.tar.gz
tar -zxf v0.25.1.tar.gz && cd libgit2-0.25.1/
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DLIB_INSTALL_DIR=/lib64
cmake --build . --target install
pip install pygit2
systemctl restart salt-master

There may have been some libcurl and libssh2 dependencies as well, but these can be satisfied from OS packages.

cedwards · 2017-02-03T01:55:59Z

Does anyone know if / where this has been submitted upstream?

terminalmage · 2017-02-03T14:34:44Z

@cedwards I referenced the bug report in an earlier comment on this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1400668

Eldrad53 · 2017-02-07T11:21:12Z

I had the same problem with salt 2015.8.12 and RedHat 7.2
After downgrade http-parser 2.7.1 to 2.0.5, GIT synchronization works correctly.

terminalmage · 2017-02-07T15:09:57Z

@Eldrad53 That might be good information to add to that upstream bug report, if it hasn't already been mentioned.

See saltstack#38630 (comment)

See #38630 (comment)

terminalmage · 2017-02-11T17:02:42Z

A build has been submitted to the Fedora Build System which should fix this issue.

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-dc9f5cde6d

The RPM for the build can be downloaded via https://koji.fedoraproject.org/koji/buildinfo?buildID=845387 and installed using rpm -Uvh filename.rpm for those who wish to test. There are warnings on import but these are due to pygit2's usage of cffi and are not fatal.

Those with Fedora accounts can give positive feedback (click the radio buttons under the check mark) to vote it into the stable EPEL repository.

terminalmage · 2017-02-14T16:04:13Z

The build I mentioned in my previous post is now in the stable repositories (pending in some cases a mirror sync):

% docker run --rm -it centos:7.3.1611
[root@e995482b4669 /]# yum -y install epel-release >/dev/null 2>&1; yum -y install python-pygit2 >/dev/null 2>&1
[root@e995482b4669 /]# rpm -q python-pygit2
python-pygit2-0.21.4-3.el7.x86_64
[root@e995482b4669 /]# rpm -qi python-pygit2 | grep 'Build Date'
Build Date  : Fri Feb 10 23:38:34 2017
[root@e995482b4669 /]# python -c 'import pygit2'
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_checkout_notify_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_merge_tree_flag_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())

The warnings there are due to version 0.21.4 of pygit2's usage of cffi, and are A) not fatal, and B) presumably fixed in later pygit2 releases.

Can someone experiencing this issue (@cedwards ?) update to 0.21.4-3 (the -3 is important, as it is the fixed build) and confirm that this resolves the issue?

timwsuqld · 2017-02-15T08:06:11Z

$ rpm -q python-pygit2
python-pygit2-0.21.4-3.el7.x86_64
$ rpm -qi python-pygit2 | grep 'Build Date'
Build Date  : Sat 11 Feb 2017 09:38:34 AEST

@terminalmage Unfortunately it doesn't seem to have fixed the problem.

Logs after salt-master restart

2017-02-15 18:04:41,306 [py.warnings      ][WARNING ][6111] /usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_checkout_notify_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())

2017-02-15 18:04:41,310 [py.warnings      ][WARNING ][6111] /usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_merge_tree_flag_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())

2017-02-15 18:04:45,211 [salt.utils.gitfs ][ERROR   ][6211] Error occurred fetching gitfs remote 'https://github.com/saltstack-formulas/remi-formula.git': No Content-Type header in response
2017-02-15 18:04:46,353 [salt.utils.gitfs ][ERROR   ][6211] Error occurred fetching gitfs remote 'https://github.com/suqld/zabbix-formula.git': No Content-Type header in response

$ salt --versions
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_checkout_notify_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
/usr/lib64/python2.7/site-packages/cffi/model.py:526: UserWarning: 'git_merge_tree_flag_t' has no values explicitly defined; next version will refuse to guess which integer type it is meant to be (unsigned/signed, int/long)
  % self._get_c_name())
Salt Version:
           Salt: 2016.11.2
 
Dependency Versions:
           cffi: 1.6.0
       cherrypy: 3.2.2
       dateutil: 2.5.3
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: 0.21.4
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.4
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.3.1611 Core
        machine: x86_64
        release: 3.10.0-514.6.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

terminalmage · 2017-02-15T16:11:47Z

@timwsuqld yeah, I can reproduce that. I did some investigation, and this is still an upstream issue. I opened a new upstream bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1422583

terminalmage · 2017-02-15T16:17:15Z

Downgrading to http-parser 2.0.5 via the RPMs at https://koji.fedoraproject.org/koji/buildinfo?buildID=703753 should allow you to proceed until such time as libgit2 is rebuilt for EPEL 7.

stale · 2018-09-08T16:26:25Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

terminalmage · 2018-09-09T01:45:14Z

The upstream bug report was confirmed fixed in April of 2017. Closing this issue.

gtmanfred added the info-needed waiting for more info label Jan 9, 2017

gtmanfred added this to the Blocked milestone Jan 9, 2017

terminalmage added Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged and removed info-needed waiting for more info labels Jan 23, 2017

terminalmage added Upstream-Bug is a result of an upstream issue, not in salt and removed Pending-Discussion The issue or pull request needs more discussion before it can be closed or merged labels Jan 24, 2017

terminalmage added a commit to terminalmage/salt that referenced this issue Feb 10, 2017

Document the upstream RedHat bug with their pygit2 package

f274220

See saltstack#38630 (comment)

terminalmage mentioned this issue Feb 10, 2017

Document the upstream RedHat bug with their pygit2 package #39316

Merged

rallytime pushed a commit that referenced this issue Feb 10, 2017

Document the upstream RedHat bug with their pygit2 package (#39316)

fa30143

See #38630 (comment)

stale bot added the stale label Sep 8, 2018

terminalmage closed this as completed Sep 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gitfs - No Content-Type header in response #38630

Gitfs - No Content-Type header in response #38630

timwsuqld commented Jan 9, 2017

gtmanfred commented Jan 9, 2017

retrry commented Jan 11, 2017 •

edited

Loading

afletch commented Jan 11, 2017

terminalmage commented Jan 23, 2017

afletch commented Jan 23, 2017

terminalmage commented Jan 23, 2017

terminalmage commented Jan 24, 2017

afletch commented Jan 25, 2017

cedwards commented Feb 3, 2017

terminalmage commented Feb 3, 2017

Eldrad53 commented Feb 7, 2017

terminalmage commented Feb 7, 2017

terminalmage commented Feb 11, 2017 •

edited

Loading

terminalmage commented Feb 14, 2017

timwsuqld commented Feb 15, 2017

terminalmage commented Feb 15, 2017

terminalmage commented Feb 15, 2017

stale bot commented Sep 8, 2018

terminalmage commented Sep 9, 2018

Gitfs - No Content-Type header in response #38630

Gitfs - No Content-Type header in response #38630

Comments

timwsuqld commented Jan 9, 2017

Description of Issue/Question

Setup

Steps to Reproduce Issue

Versions Report

gtmanfred commented Jan 9, 2017

retrry commented Jan 11, 2017 • edited Loading

Versions Report

afletch commented Jan 11, 2017

terminalmage commented Jan 23, 2017

afletch commented Jan 23, 2017

terminalmage commented Jan 23, 2017

terminalmage commented Jan 24, 2017

afletch commented Jan 25, 2017

cedwards commented Feb 3, 2017

terminalmage commented Feb 3, 2017

Eldrad53 commented Feb 7, 2017

terminalmage commented Feb 7, 2017

terminalmage commented Feb 11, 2017 • edited Loading

terminalmage commented Feb 14, 2017

timwsuqld commented Feb 15, 2017

terminalmage commented Feb 15, 2017

terminalmage commented Feb 15, 2017

stale bot commented Sep 8, 2018

terminalmage commented Sep 9, 2018

retrry commented Jan 11, 2017 •

edited

Loading

terminalmage commented Feb 11, 2017 •

edited

Loading