Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot parse OpenSSH 6.5 formatted private keys #602

Open
kabel opened this issue Oct 28, 2015 · 21 comments
Open

Cannot parse OpenSSH 6.5 formatted private keys #602

kabel opened this issue Oct 28, 2015 · 21 comments
Labels

Comments

@kabel
Copy link

@kabel kabel commented Oct 28, 2015

The key format described at https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.key is not currently supported causing key authentication problems when that format is used. The current parser erroneously assumes that the key file is unencrypted.

Related to #387

@mchlt

This comment has been minimized.

Copy link

@mchlt mchlt commented Nov 5, 2015

Looking into this issue.
Some additional info about the key format here:
http://www.tedunangst.com/flak/post/new-openssh-key-format-and-bcrypt-pbkdf

@mchlt mchlt mentioned this issue Nov 7, 2015
@mchlt

This comment has been minimized.

Copy link

@mchlt mchlt commented Nov 7, 2015

OK, i now have some working code that correctly reads the base64 blobs from a new OPENSSH type key.
Unfortunately, OpenSSH have chosen a rather exotic key derivation function to derive the key from the passphrase: bcrypt_pbkdf
The only Python implementation of that that I could find is in py_bcrypt (https://github.com/grnet/python-bcrypt). I'll use that for now to see if I can actually do the decryption.

Wondering what's paramiko's stance on introducing new module dependencies? I'm thinking maybe do the import in a try..except and then throwing an exception only if someone tries to load an encrypted new format key?

@bitprophet

This comment has been minimized.

Copy link
Member

@bitprophet bitprophet commented Nov 9, 2015

@mchlt We do the "try/except/complain" for other optional deps right now, such as python-gssapi for GSSAPI support (docs). So we don't want to do it willy-nilly, but there's precedent.

That said, "a new privkey format used by OpenSSH" feels more mainstream a feature (at least going forwards) than Kerberos/GSSAPI support, so I lean towards "if we can get this implemented before I finish the other work for 2.0, let's put python-bcrypt in that release as a hard dependency".

If you haven't, you might also look to see if https://github.com/pyca/cryptography has it (this mentions pbkdf2 but I'm not crypto-fluent enough to tell if it's truly the same as python-bcrypt's C-ext) or is willing to add it, since paramiko 2.0 will be relying on cryptography anyways.

@bitprophet

This comment has been minimized.

Copy link
Member

@bitprophet bitprophet commented Nov 9, 2015

Also, thanks a lot for looking into this! ❤️

@bitprophet

This comment has been minimized.

Copy link
Member

@bitprophet bitprophet commented Apr 25, 2016

Continuing discussion in here since not 100% sure what'll happen to #618 (it might want closing & replacing with another PR):

  • #618 notes that Cryptography doesn't support bcrypt (specifically bcrypt_pbkdf). I'm curious whether @alex / @reaperhulk think it's worth adding to Cryptography at any point? Don't see anything over there implying such, but always nice to ask.
  • Assuming that's not in the works, I'm still reasonably okay with adding the python-bcrypt dependency instead. (TBH, even if Cryptography added it I wouldn't be surprised if they wrapped python-bcrypt itself, so...)
  • In either case, however, #618 doesn't merge cleanly with #394 which is about to land in master.
    • If @mchlt has time in the very near future to address that (ideally a few days; yes, asking another to get back to me in days when I tend to get back in months, is horrid 😖 ), I can squeeze this into 2.0 as originally planned, and add python-bcrypt as another new install-time dependency.
    • If he doesn't (which would be fine!) I'll put it into the 3.0 bucket, which is basically what 2.0 used to be.
      • That will still likely come out sooner instead of later (again see #394 for thoughts on release junk), FYI - not planning it to be another very long term major release but a short/mid term one.
@reaperhulk

This comment has been minimized.

Copy link
Contributor

@reaperhulk reaperhulk commented Apr 25, 2016

We'd like to support bcrypt directly, but right now cryptography is (mostly) tied to OpenSSL which doesn't support eksblowfish. We do have pyca/bcrypt, which will shortly be getting wheels for OS X and even linux (via manylinux1, see PEP 513).

@bitprophet

This comment has been minimized.

Copy link
Member

@bitprophet bitprophet commented Apr 25, 2016

@reaperhulk Huh, neat - I'd still feel better about depending on a PyCA-blessed/implemented bycrypt module vs one that seems to largely be a one-off (not to diss python-bcrypt / py-bcrypt of course, but.) Thanks!

/cc @mchlt - if pyca/bcrypt serves your needs as well as the other one, maybe switching to it would be something to do while updating.

@mchlt

This comment has been minimized.

Copy link

@mchlt mchlt commented Apr 25, 2016

Whoa, all going fast suddenly :)
Will try to see what I can do tonight

Unfortunately pyca/bcrypt does not have the bcrypt pbkdf function which is
needed for the new openssh private key file format.

Op ma 25 apr. 2016 02:19 schreef Jeff Forcier notifications@github.com:

@reaperhulk https://github.com/reaperhulk Huh, neat - I'd still feel
better about depending on a PyCA-blessed/implemented bycrypt module vs one
that seems to largely be a one-off (not to diss python-bcrypt / py-bcrypt
of course, but.) Thanks! /cc @mchlt https://github.com/mchlt - if
pyca/bcrypt serves your needs as well as the other one, maybe switching to
it would be something to do while updating.


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#602 (comment)

@reaperhulk

This comment has been minimized.

Copy link
Contributor

@reaperhulk reaperhulk commented Apr 25, 2016

@mchlt Interesting, we definitely don't. I'll take a look at what it would require to do that today. Unfortunately the underlying C lib we use (crypt_blowfish) hasn't been updated to add support for that.

@reaperhulk

This comment has been minimized.

Copy link
Contributor

@reaperhulk reaperhulk commented Jul 9, 2016

As an update, PyCA bcrypt 3.1.0 is out which provides a bcrypt_pbkdf and ships wheels for Windows, Mac, and manylinux1 so in many cases you won't need a compiler at all!

@ploxiln

This comment has been minimized.

Copy link
Contributor

@ploxiln ploxiln commented Jun 7, 2017

Support for the new bcrypt-protected key format was added, just for ed25519 keys, in #972

(support is still missing for rsa keys in the new bcrypt-based format - I actually use such a key, but via an ssh-agent)

@trou

This comment has been minimized.

Copy link

@trou trou commented Aug 22, 2018

Just chiming in for the new RSA key format, it would be nice to have it, as it is way more secure that the old format.

@raoulh

This comment has been minimized.

Copy link

@raoulh raoulh commented Sep 27, 2018

Paramiko needs this new key format as Openssh defaults to this format since the latest release.

https://www.openssh.com/releasenotes.html

  • ssh-keygen(1): write OpenSSH format private keys by default
    instead of using OpenSSL's PEM format. The OpenSSH format,
    supported in OpenSSH releases since 2014 and described in the
    PROTOCOL.key file in the source distribution, offers substantially
    better protection against offline password guessing and supports
    key comments in private keys.
openstack-gerrit pushed a commit to openstack/sahara that referenced this issue Sep 28, 2018
Unfortunately it is not possible to switch to the new, more secure, native
format of OpenSSH >=6.5, because paramiko does not support it:
paramiko/paramiko#602

This change should fix the unit test (and probably the behavior)
when sahara services are executed on distributions which ships
OpenSSL 1.1 and which switched to the new format by default
(at least the current Debian Sid and Fedora 28).

Story: 2003674
Task: 26193
Change-Id: I51c8daebe42345ee5d610356d2c1710a069f0355
openstack-gerrit pushed a commit to openstack/sahara that referenced this issue Oct 1, 2018
Unfortunately it is not possible to switch to the new, more secure, native
format of OpenSSH >=6.5, because paramiko does not support it:
paramiko/paramiko#602

This change should fix the unit test (and probably the behavior)
when sahara services are executed on distributions which ships
OpenSSL 1.1 and which switched to the new format by default
(at least the current Debian Sid and Fedora 28).

Story: 2003674
Task: 26193
Change-Id: I51c8daebe42345ee5d610356d2c1710a069f0355
(cherry picked from commit e0fd845)
tellesnobrega pushed a commit to tellesnobrega/sahara-plugins that referenced this issue Oct 1, 2018
Unfortunately it is not possible to switch to the new, more secure, native
format of OpenSSH >=6.5, because paramiko does not support it:
paramiko/paramiko#602

This change should fix the unit test (and probably the behavior)
when sahara services are executed on distributions which ships
OpenSSL 1.1 and which switched to the new format by default
(at least the current Debian Sid and Fedora 28).

Story: 2003674
Task: 26193
Change-Id: I51c8daebe42345ee5d610356d2c1710a069f0355
@TurboTurtle

This comment has been minimized.

Copy link

@TurboTurtle TurboTurtle commented Oct 30, 2018

Just checking in here, is the issue with #618 still the same as noted in the April 24 2016 comment above?

Is there any timeframe on paramiko supporting the new key format?

@jaredhobbs

This comment has been minimized.

Copy link
Contributor

@jaredhobbs jaredhobbs commented Nov 29, 2018

I have a PR open #1343 based on the code in #618 that will merge cleanly with master.

@OddBloke

This comment has been minimized.

Copy link

@OddBloke OddBloke commented Dec 12, 2018

The OpenSSH key format became the default generated by ssh-keygen in version 7.8, so people are going to start hitting this issue more and more in the wild (without having explicitly opted in to the new format).

In my specific case, Ubuntu 19.04 (Disco Dingo), the currently in-development release of Ubuntu, includes a post-7.8 version (7.9p1 at the time of writing). Ubuntu users who upgrade in April are going to start hitting this issue.

openstack-gerrit added a commit to openstack/openstack that referenced this issue Jan 7, 2019
* Update sahara from branch 'master'
  - APIv2 - api-ref documentation for APIv2
    
    Writing api-ref for APIv2.
    
    Story: #2002102
    Task: #19780
    
    Change-Id: Ib9e855c11f03239e70306d12e96194549d2dc0f3
    
  - Merge "APIv2 Changing return payload to project_id"
  - Fix validation of job binary with Python3
    
    Currently with Python3 it raises exception
    BadJobBinaryInternalException() with Data: b'test-project',
    This patch handles it for python3.
    
    Change-Id: I45ce95fdec78af7f21a98613e8c7763bd84ff2aa
    Story: 2004688
    Task: 28698
    
  - Merge "Bump the version of hacking to 1.1.0, with few fixes"
  - Merge "String-related fixes for Python 3"
  - Merge "doc: Fix the snippet in "The Script Validator" section"
  - APIv2 Changing return payload to project_id
    
    As part of the APIv2 work we need to change all tenant_id references
    to project_id on the return payload.
    
    Story: #2004505
    Taks: #28227
    
    Change-Id: I94bca161aa4f7bdd56d5528bae92fa81af188a43
    
  - Fixing cluster scale
    
    The current implementation fails if we try to scale from different
    node groups.
    
    Change-Id: Ifb9e4b55959e10c9e5cb74c86bbdba9ffed50ceb
    
  - doc: Fix the snippet in "The Script Validator" section
    
    In the snippet of code, store_nfs_version should point to an object
    instead of an array.
    
    Change-Id: I5093baf6fa849acba0dcacdc813ec22f01c35a84
    
  - String-related fixes for Python 3
    
    - Check if a variable is a strings using isinstance and six.
      At least one of the two checks before the fix triggers
      an exception visible in the logs when using internal
      job binaries ("Job binary internal data must be a string of
      length greater than zero").
    - Encode a string that is passed to md5, as the error suggests
      ("Unicode-objects must be encoded before hashing")
    
    Change-Id: Icb5d75bdbfb83070c579b9b99a395f344c3120ce
    
  - Merge "Update devel info: mailing list, meeting time"
  - Merge "fixed word error"
  - fixed word error
    
    Change-Id: I83f8abacbc1125f688daa61178a1d107f61f1dba
    
  - Add DEBIAN_FRONTEND=noninteractive in front of apt-get install commands
    
    The goal is to avoid a failure of apt-get install if one of the packages
    tries to ask questions.
    
    Story: #2004468
    Task: #28158
    Change-Id: I258d5c904c29110ccdb3a7fdff5b69f489552063
    
  - Bump the version of hacking to 1.1.0, with few fixes
    
    Also switch to pycodestyle from pep8 which requires a bump
    of the the version of flake8 too.
    
    Skip the following checks for now:
    - E123 Closing bracket does not match indentation of opening bracket's line
    - E226 Missing whitespace around arithmetic operator
    - E402 Module level import not at top of file
    - E731 Do not assign a lambda expression, use a def
    - W503 Line break occurred before a binary operator
    
    They should be probably revisited and fixed, if possible,
    or skipped individually using # noqa
    
    The following checks reported a limited number of errors and
    they were fixed directly in this review:
    - E241 Multiple spaces after ','
    - E501 Line too long (82 > 79 characters)
    
    Change-Id: I1f185d2efd1adf27a26e3ac93f2e1011c0b63124
    
  - Merge "Add python 3.6 unit test job"
  - Merge "Update http link to https"
  - Update devel info: mailing list, meeting time
    
    - The new openstack-discuss mailing list is going to replace few
      mailing lists, including openstack-dev.
    - There is only one meeting time now (1400UTC).
    
    Change-Id: I14c5d37ae7f59095d00a7b26fa0a0fc23b01a33a
    
  - Merge "Increase the startup time of ambari-server to 180s"
  - Update http link to https
    
    Modify http link to https link
    
    Change-Id: I73517f80361b12da09baac5b627d580a9c9f4295
    
  - Add python 3.6 unit test job
    
    This is a mechanically generated patch to add a unit test job running
    under Python 3.6 as part of the python3-first goal.
    
    See the python3-first goal document for details:
    https://governance.openstack.org/tc/goals/stein/python3-first.html
    
    Change-Id: Ia9927693c1ef48e04b8eaa8d0754ce294f41cd97
    
  - Merge "Add framework for sahara-status upgrade check"
  - Add framework for sahara-status upgrade check
    
    This commit adds the functionality of sahara-status CLI for performing
    upgrade checks as part of the Stein cycle upgrade-checkers goal.
    It only includes a sample check which must be replaced by real checks in
    future.
    
    Change-Id: Idcb8d9eaf689800812cf6087e9c5937058c89ea6
    Story: 2003657
    Task: 26152
    
  - doc: restructure the image building documentation
    
    Main goal: consolidate the information about image
    building under the same documentation page, and move
    plugin-specific details inside plugin pages.
    No plugin-specific information should live outside
    those pages.
    
    More details:
    - move the detailed documentation about sahara-image-pack
      from the contributor guide to the new dedicated page
      in the user manual;
    - remove the vanilla and cdh pages which describes building
      images with sahara-image-create, and move the common
      information to new sahara-image-create page
      in the user manual;
    - add the matrix of supported plugin versions and
      supported building technology for each plugin inside
      the respective <plugin>-plugin.rst;
    - add the redirects for the removed pages (only for master
      and rocky, where this change should be backported).
    - remove few details not really needed (e.g. how to convert
      to VMDK images, location of cloud-init packages, etc,
      which do not really belong here).
    
    Change-Id: I8398a7ad625276d8f11d743688ba71902a7e1adc
    
  - Merge "Fixing image validation for Ambari 2.3"
  - Fixing image validation for Ambari 2.3
    
    Changing ambari version for image validation.
    
    Story: #2003996
    Task: #26942
    Change-Id: I54a1370c482a3a2862f1c8313a984fece25efbd2
    
  - Cleanup tox.ini constraint handling
    
    Use the "modern" way of contraints setup and remove double setting of
    constraints (in install_command and deps) for some environments.
    
    Remove also -U from pip install command, it can break with constraints
    and update required packages.
    
    Change-Id: I2412a02dcba40a3128f9af766e27c046ce3d3f25
    
  - Increase the startup time of ambari-server to 180s
    
    It seems to be easy to hit timeout errors, so let's increase
    the value of the timeout when ambari-server is started.
    The option is recognized only from Ambari 2.5 onwards, but
    the plugin uses Ambari 2.6 for all versions but HDP 2.3.
    
    Story: 2004102
    Task: 27509
    Change-Id: I47c6dd14585ed20998fcf4a068cf1e7144eaf026
    
  - Increment versioning with pbr instruction
    
    With moving away from required milestone releases, the version numbers
    calculated by PBR on the master branch will not work for those testing
    upgrades from the last stable release. More details can be found in the
    mailing list post here:
    
        http://lists.openstack.org/pipermail/openstack-dev/2018-October/135706.html
    
    This is an empty commit that will cause PBR to increment its calculated
    version to get around this.
    
    PBR will see the following which will cause it to increment the version:
    
    Sem-Ver: feature
    
    Please merge this patch as soon as possible to support those testing
    upgrades.
    
    Change-Id: I399dafd4e6b4f5f906d489f225d4f3ba9bc8113a
    Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
    
  - Merge "Fix a typo on Storm plugin cluster info (Strom -> Storm)"
  - Fix a typo on Storm plugin cluster info (Strom -> Storm)
    
    Change-Id: I03a11ce2e88ad895504301c00c6ddd67984e1021
    
  - sahara-image-pack: use curl for tarballs.openstack.org
    
    It seems that wget throws some errors from time to time when getting
    content from tarballs.openstack.org.
    The errors are SSL-related, maybe some interaction between the SSL libaries
    of the libguestfs appliance and the load balancing configuration
    of tarballs.openstack.org (SNI-based).
    curl seems to be immune, so let's use it.
    
    Story: 2003961
    Task: 26890
    Change-Id: Ia5793a4be8cde352329e861c46de8c4eeac4312e
    
  - sahara-image-pack: remove bashisms from shell scripts
    
    During image generation, some shell scripts are run inside the image
    (e.g. wget_repo). Those scripts are run with /bin/sh but some parts are
    written with Bash-specific syntax. This commit replaces those parts by
    sh-compliant syntaxes.
    
    Change-Id: If64b85c7b026a70c99190a2cb6487e3f1b865928
    Story: #2003893
    Task: #26767
    Task: #26776
    
  - Merge "Force the format of ssh key to PEM, at least for now"
  - adds unit test for ssh_remote.replace_remote_line
    
    Adding unit test for the new method added to
    search a line by string and replace the old line
    with a new line
    
    Change-Id: I95d71bafebd9d0a4fea499813135fac06d152ab6
    Story: #2003176
    Task: #26708
    
  - Force the format of ssh key to PEM, at least for now
    
    Unfortunately it is not possible to switch to the new, more secure, native
    format of OpenSSH >=6.5, because paramiko does not support it:
    https://github.com/paramiko/paramiko/issues/602
    
    This change should fix the unit test (and probably the behavior)
    when sahara services are executed on distributions which ships
    OpenSSL 1.1 and which switched to the new format by default
    (at least the current Debian Sid and Fedora 28).
    
    Story: 2003674
    Task: 26193
    Change-Id: I51c8daebe42345ee5d610356d2c1710a069f0355
    
  - Merge "Add template param for ambari pkg install timeout"
  - Add template param for ambari pkg install timeout
    
    Often time ambari fails during cluster installation/service
    starting stage. This is quiet prominent when  there is
    a large number of nodes in the cluster. Review of the
    logs from the cluster indicates that ambari installation
    scripts has a timeout parameter set to 1800 sec, this
    requires adjustment depending on the environment and
    speed of package installation.
    
    This fix provides one parameter named
    "agent.package.install.task.timeout"  inside the Ambari tab
    of the HDP cluster template UI . User may change the
    values and accordingly the ambari server will be setup
    for package installation timeout.'
    
    Change-Id: I826dbebb446d49e01e3cd6d7e525b43aa4523434
    Story: #2003176
    Task: #23320
    
  - Merge "Use templates lower-constraints, update cover job"
  - Merge "grenade: relevant fixes for master (sahara-api/apache)"
  - Use templates lower-constraints, update cover job
    
    Use openstack-lower-constraints-jobs template.
    
    Remove jobs that are part of the templates.
    
    Remove post job, this is not checked by anybody. The current job is
    setup as check job and compares to last change, so no need for a post
    job - especially for a post job that nobody checks.
    
    Change-Id: I4fefd5b29c1b16b886c72fc42fbedc07d82397c4
    
  - grenade: relevant fixes for master (sahara-api/apache)
    
    - include lib/apache when appropriate;
    - use devstack@sahara-eng for sahara-engine;
    - do not check for sahara-api, as the stop/start_sahara code should
      take care of it (both for the old case and the wsgi case).
    
    Change-Id: I22bb6d51551b6c2540de09e7f307cef27aba0f23
    
  - doc: update distro information and cloud-init users
    
    - centralize the information about the default cloud-init users
      for various distributions to point to the existing specialized
      page;
    - remove all CentOS 6 and Ubuntu 12.04 (Precise) references, and
      do not refer to Ubuntu 14.04 for the vanilla plugin;
    - add Ubuntu 16.04 (Xenial) details when needed;
    - update the pointers to few packages.
    
    Change-Id: I4654e6ec2821bcec5207f41b93c5d0f0633ba18c
    
  - Fixed link for more information about Ambari images
    
    Change-Id: I91a3310dfff2687975cb8c7c7693106b509943e7
    
  - Merge "Correct repo_id_map for hdp 2.5"
  - Correct repo_id_map for hdp 2.5
    
    When hdp2.5 cluster installation is done with off-line
    repo,hdp-utils repo name should be HDP-UTILS-1.1.0.21.
    However this is hardcoded wrongly in sahara repo_id_map
    as HDP-UTILS-1.1.0.20.As a result Ambari HDP repo set up
    fails.
    This fix will correct the repo_id_map
    
    Change-Id: Ibf7d341cc2d2a53be521039d4f843cdbc5ee880b
    Story: #2003654
    Task: #26067
    
  - Make sahara-grenade job voting on the "gate" queue too
    
    It should have been done in the previous commit, will be squashed
    with the main import change when backporting to the older branches.
    
    Change-Id: Icda0eb2f0b001f4468ba86af65b8c5c12fbaebab
    
  - Import the legacy grenade sahara job
    
    While it would have been better to directly use a native Zuul v3 job,
    the work towards a common grenade job seems to be stalled.
    Import the job locally and fix some issues:
    - make sure that the heat_tempest_plugin is loaded correctly,
      so that the heat upgrade tests work;
    - bump the starting point from stable/ocata (!) to stable/queens;
      it will be changed to stable/rocky when grenade supports it;
      will require additional fixes due to the standalone->uwsgi change.
    - import lib/apache so that wsgi deployment works.
    
    Story: 2001686
    Task: 24517
    Change-Id: Ia8de2b8286287355e8e9d3f702027aa282ed8282
    
  - Correct Hbase ports in Ambari plugin
    
    port number for HBase configured in Sahara starts with 600xx
    where as hortonworks ambari configures these ports on the
    hbase master and region servers with 160xx.Although Ambari
    will start the master server with the 160xx port, clients
    can't connect them due to improper security rules.As a
    result hbase master web info port link in the cluster
    general info page doesn't work.HBase region server won't start.
    
    This patch sets the HBase ports correctly for Ambari Plugin.
    
    Change-Id: Ic13944ed729c0840578784f50a53f17b0706b62c
    Story: #2003562
    Task: #24853
    
  - Merge "Adapt to Keystone changes: use member instead of Member"
  - Merge "Fixing anti-affinity for Sahara"
  - Fixing anti-affinity for Sahara
    
    Sahara anti-affinity was broken since some problematic changes in
    cd1569852614698c4843d4c97475d8f8f3069478.
    
    This should be able to fix it.
    
    Change-Id: I374c0340cb0f85c00b9a04cd1b23e3912737994c
    Co-Authored-By: Joe Topjian <joe@topjian.net>
    Story: #2002656
    Task: #22466
    
  - add python 3.6 unit test job
    
    This is a mechanically generated patch to add a unit test job running
    under Python 3.6 as part of the python3-first goal.
    
    See the python3-first goal document for details:
    https://governance.openstack.org/tc/goals/stein/python3-first.html
    
    Change-Id: I8bd10600daa79cb92b8be35fa4ddd708231f569d
    Story: #2002586
    Task: #24332
    
  - switch documentation job to new PTI
    
    This is a mechanically generated patch to switch the documentation
    jobs to use the new PTI versions of the jobs as part of the
    python3-first goal.
    
    See the python3-first goal document for details:
    https://governance.openstack.org/tc/goals/stein/python3-first.html
    
    Change-Id: Ibb49209b4c8923a51c53ebaf0fdddb10eed8addf
    Story: #2002586
    Task: #24332
    
  - import zuul job settings from project-config
    
    This is a mechanically generated patch to complete step 1 of moving
    the zuul job settings out of project-config and into each project
    repository.
    
    Because there will be a separate patch on each branch, the branch
    specifiers for branch-specific jobs have been removed.
    
    Because this patch is generated by a script, there may be some
    cosmetic changes to the layout of the YAML file(s) as the contents are
    normalized.
    
    See the python3-first goal document for details:
    https://governance.openstack.org/tc/goals/stein/python3-first.html
    
    Change-Id: Ic482013bf68ab2f21cdb6c1a4a5c4d6ee5b05788
    Story: #2002586
    Task: #24332
    
  - Merge "Update reno for stable/rocky"
  - Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: Id232e0d13db035594b7458f0b2ae789659413890
    
  - Update reno for stable/rocky
    
    Change-Id: I80c274e70512bd6088f14572e3c6b03f00b63262
    
  - Merge "Add some S3 doc"
  - Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I7248fc6007eb179413200e0d92276553ca094ba4
    
  - Merge "Enable also ambari by default in devstack"
  - Adapt to Keystone changes: use member instead of Member
    
    Keystone now provides "member" by default.
    It should not be a problem for upgrades when the configuration
    is the default one, because Keystone is case-preserving and
    Member and member are the considered the same:
    https://docs.openstack.org/keystone/latest/admin/identity-case-insensitive.html
    
    Change-Id: I3bd72631d57546dcf8b887833539fe3ccaac4e47
    
  - Add some S3 doc
    
    Perhaps some more to follow, someday, but this is nice to have.
    
    Change-Id: I2235f903105049432de24d89a88b40f753fd93d6
    
  - Merge "Another small fix for cluster creation on APIv2"
  - Merge "S3 data source URL format change"
  - Enable also ambari by default in devstack
    
    It should not impact the usual features of devstack,
    but no reason to not enable it.
    The old hdp plugin was removed but this one was never added.
    
    Change-Id: Ie93948966b90e286eac401673e6704694bacf249
    
  - Merge "Sets correct permission for /etc/hosts"
  - Another small fix for cluster creation on APIv2
    
    Small fix that was blocking creation of multiple clusters using api v2.
    
    Change-Id: I6c1db62a4bee3d1b064333b7b2d8b8e2d6ad50f1
    
  - S3 data source URL format change
    
    The old way will still work, but prefer s3:// now.
    
    Change-Id: Ia1f8eba22016044aa5ffe50b2ab898908aef1890
    
  - Sets correct permission for /etc/hosts
    
    presently while genearting the hosts file and
    replacing the original one , the file permission is
    replaced  with no access for world and ownership
    changes from root to centos.
    
    Secondary nodes, in the cluster while trying to
    resolve primary host address can't  read the host
    name and ip from hosts file and fails to connect.
    
    This fix restores the hosts file permission correctly
    
    Change-Id: Ia9b7e609644c93d13108af88aa3dacf94fc19ceb
    Task: #23261
    Story: #2003139
    
  - Merge "Allow overriding of /etc/hosts entries"
  - Fixing cluster creation on APIv2
    
    We missed this detail on the review. Right now the creation of a single cluster
    fails.
    
    Change-Id: I56a94d9045284259b0f0a8b3998106b754187c4f
    
  - Allow overriding of /etc/hosts entries
    
    Sometimes the DNS lookup performed where Sahara services are running
    may be inaccurate.
    
    Change-Id: I27d2bc4ad9340440878a2f342549d7fe74ee7e68
    Story: 2001737
    Task: 12092
    
  - Merge "Enable mutable config in sahara"
  - Enable mutable config in sahara
    
    New releases of oslo.config support a 'mutable' parameter to Opts.
    oslo.service provides an option here Icec3e664f3fe72614e373b2938e8dee53cf8bc5e
    allows services to tell oslo.service they want mutate_config_files to be
    called by passing a parameter.
    
    This commit is to use the same. This allows sahara to benefit from
    I1e7a69de169cc85f4c09954b2f46ce2da7106d90, where the 'debug' option
    (owned by oslo.log) is made mutable. we should be able to turn debug
    logging on and off by changing the config.
    
    tc goal:
    https://governance.openstack.org/tc/goals/rocky/enable-mutable-configuration.html
    
    Change-Id: I11d31a80afb326fa3416726c6f68b4930346264a
    
  - Merge "Unversioned endpoint recommendation"
  - Merge "Replace the deleted keypair in clusters for API v2"
  - Merge "Adding Ambari 2.6 to image pack"
  - Merge "Boot from volume"
  - Merge "api-ref: move to a v1.1 sub-folder"
  - Merge "Trivial: Update Zuul Status Page to correct URL"
  - Merge "Adding Storm 1.2.0 and 1.2.1"
  - Adding Ambari 2.6 to image pack
    
    Adding ambari 2.6 to image pack so users can create image to spawn
    HDP 2.6
    
    Change-Id: Ifb90d8ba1958403f2c00eade013e147660357717
    
  - Adding Storm 1.2.0 and 1.2.1
    
    Adding new versions of Storm to sahara.
    
    Change-Id: I7f4a96f2dc8cb66468866f77e3d4091d2a8d19d1
    
  - Unversioned endpoint recommendation
    
    APIv2 is reaching stability. People will want to use it. It's best,
    although not actually required, if deployers put an unversioned
    endpoint for Sahara in the service catalog.
    
    Change-Id: I0256c2002c6c25969fd23cc6f6d3b11c3e577646
    
  - api-ref: move to a v1.1 sub-folder
    
    Make space for v2.
    It should not break too much existing links, only the links to
    local anchors (like #clusters, #jobs, etc) will be broken,
    because it is not possible to redirect server-side based on an
    URL fragment like that.
    Luckily the generated content ends all in the same HTML page,
    so the old links will end up in the new top-level index
    which points to the index for v1.1 API documentation (and later
    for v2). Not too bad.
    
    Change-Id: I9a75e6567798b8d6d3deed8fc66e362ad2d86b5a
    
  - Merge "Switch make_json_error back to being a function"
  - Merge "Updating Spark versions"
  - Merge "Switch the coverage tox target to stestr"
  - Merge "S3 data source"
  - Trivial: Update Zuul Status Page to correct URL
    
    Current URL of Zuul Status Page in code is:
    http://status.openstack.org/zuul/
    
    The correct URL must be:
    https://zuul.openstack.org/
    
    Remove outdated Jenkins reference.
    
    Change-Id: I7119fe6818a2a4b3144b5cd8b2d241ff8f2cdbb2
    
  - Switch make_json_error back to being a function
    
    This was originally a function, and then I changed it to a dict because
    Flask complained. Now Flask complains that is a dict, so change it back
    to being a function.
    
    Change-Id: Iec30b4cf5023be711ac070def00b77e91978d992
    
  - Merge "Final fixup to APIv2 responses"
  - Merge "Deprecate sahara-all"
  - Final fixup to APIv2 responses
    
    Some inconsistencies still remained.
    
    Change-Id: Ic54af5433f6498488f457d8e49dae74e684e2bd2
    Story: 1745294
    Task: 8392
    
  - Deprecate sahara-all
    
    Change-Id: I7585193d00e933a4fce556b9cd5e62fef8d1dd83
    
  - Switch hive_enable_db_notification's default value
    
    Make this change for CDH 5.13.0.
    
    And do so, such that the user may...
    * easily observe the new default value
    * easily override the new default value
    
    Change-Id: Ic2805ee0fb6bc2d96144e86b8a3f49beae3e1cef
    
  - S3 data source
    
    * Create S3 data source type for EDP
    * Support storing S3 secret key in Castellan
    * Unit tests for new data source type
    * Document new data source type and related ideas
    * Add support of S3 configs into Spark and Oozie workflows
    * Hide S3 credentials in job execution info, like for Swift
    * Release note
    
    Change-Id: I3ae5b9879b54f81d34bc7cd6a6f754347ce82f33
    
  - Switch the coverage tox target to stestr
    
    Use coverage+stestr for the cover tox target.
    With this change, testrepository is not required anymore.
    
    Change-Id: Ia294ee8003ccc10104f68777bea6921ff97bacef
    
  - Merge "Fixing extjs check on cdh and mapr"
  - Merge "Switch ostestr to stestr"
  - Merge "Bump Flask version according requirements"
  - Updating Spark versions
    
    We are adding new spark version 2.3.0
    
    Change-Id: I3a1c8decdc17c2c9b63af29ee9199cf24f11e0e2
    
  - Fixing extjs check on cdh and mapr
    
    On images generated from sahara-image-elements cluster is being stuck
    due to double execution of the unzip code.
    
    This fix prevents the code to be executed twice and in the case it does we
    force unzip to overwrite.
    
    Change-Id: I73836a516b839bdb368997af3693c139c1fff390
    Story: #2002551
    Task: #22113
    
  - Merge "Fix flask.request.content_length is None"
  - Switch ostestr to stestr
    
    According to Openstack summit session [1] stestr is
    maintained project to which all Openstack projects
    should migrate.
    Let's switch it then.
    
    [1] https://etherpad.openstack.org/p/YVR-python-pti
    
    Change-Id: Ic580dc7ba0684087a81137771fb125f4d412d948
    Signed-off-by: Charles Short <zulcss@gmail.com>
    
  - Bump Flask version according requirements
    
    The mimimum requirement of Flask was recently bumped to 1.0.2
    (https://review.openstack.org/#/c/577534/) which means that
    the requirements-check job used for the sahara gate
    is failing with a requirement mismatch errors.
    
    Change-Id: I0fc7e3e8a847917be0877f71128a603d258a85ea
    
  - Merge "Remove any reference to pre-built images"
  - Fix flask.request.content_length is None
    
    In Python 3 (I haven't checked with Python 2), it appears that sahara-api
    is broken, checking for flask.request.content_length > 0, when this
    object doesn't exist, and then it fails with:
    
    TypeError: unorderable types: NoneType() > int()
    
    This is due to differences between py2 and py3.
    
    This patch therefore tests if flask.request.content_length is None in the
    validation code.
    
    Change-Id: Iedddc1b2a8d5c3d88d02a380eaf85206a234494a
    
  - Merge "Deploy using wsgi by default"
  - Merge "Updating plugins status for Rocky"
  - Merge "Better default value for domain in swift config"
  - Use register_error_handler to register make_json_error
    
    error_handler_spec[None] doesn't exist by default in flask
    versions >= 1.0. It doesn't appear to be the right way to
    register an error handler anyway, assuming that's what it is
    doing. Use register_error_handler instead.
    
    Change-Id: I6a56e1edf277c5652a876658f8cfbb5550a73ada
    Story: 2002617
    Task: 22246
    
  - Boot from volume
    
    Adding the ability to boot a sahara cluster from volume.
    
    Story: #2001820
    Task: #12558
    
    Change-Id: Ie11c5e7a628c369868d3c56e803da4b9e7d15f85
    
  - Remove any reference to pre-built images
    
    This removes the very last references to sahara-files.mirantis.com,
    which can be discontinued any time soon. Thanks Mirantis for the
    hosting so far!
    There are currently no plan to offer those pre-built images.
    
    Story: 2001996
    Task: 19628
    Change-Id: I6b362c6f630495b16bc9a1e2e1b7ebbf4405ef8a
    
  - Updating plugins status for Rocky
    
    For Rocky we are removing some plugins versions:
    
    - CDH: removing 5.5.0
    - MapR: removing 5.1.0
    - Spark: removing 1.3.1
    - Storm: removing 0.9.2
    
    Also, we are marking some versions as deprecated:
    
    - CDH: deprecating 5.7.0
    - Spark: deprecating 1.6.0 and 2.1
    - Storm: deprecating 1.0.1
    
    Change-Id: I2dcec1344db4225263be179366eb408d62b9e108
    
  - Replace the deleted keypair in clusters for API v2
    
    This commit allows users to create a new keypair to access to
    the running cluster when the cluster's keypair is deleted. But
    the name of new keypair should be same as the deleted one, and
    the new keypair is available for cluster scaling.
    we can implement it via update_keypair parameter.
    
    Change-Id: I3bc0ba52ceebff8ae128321aeb9fdbb5769baaf5
    Closes-Bug:#1745142
    
  - Better default value for domain in swift config
    
    The default value of domain id as 'default' set in the Java code is not
    always ideal.
    
    This is not totally ideal either (see the comments) but helps somewhat.
    
    Change-Id: Ib5433ee441fb4d63f76e43e0c449e97d197a5b18
    
  - Fix the code repository for clone action
    
    Change-Id: Id6c1946c193a6d3668ea0885697ba03d1dacc238
    
  - add release notes to readme.rst
    
    Change-Id: I02f0abb318f70df2d269b0c3494ee6d6882333c1
    
  - Merge "Adding CDH 5.13"
  - Adding CDH 5.13
    
    We are adding a new version of CDH to the list of supported plugins.
    
    Change-Id: Ia55c6729dc6c4640b83e1d2d4dba88d8bba29e36
    Story: #2002183
    Task: #20056
    
  - Merge "Improve force delete"
  - Improve force delete
    
    * Drop use of stack abandon: just use a regular delete instead
    * Return stack name from force delete API call
    
    Change-Id: I33ee7323fade1b237957abb8f7c79b87eb20148f
    
  - Merge "doc: light cleanup of the ironic-integration page"
  - Merge "Updated oozie version"
  - Updated oozie version
    
    Old download link expired
    
    Change-Id: I7c98a14511350f53d271ddec62b3d76665ff28a7
    
  - Merge "doc: external link helper for other projects' doc"
  - Merge "Check node processes earlier"
  - Merge "[APIv2]Consolidate cluster creation endpoints"
  - Merge "Update the command to change the hostname"
  - doc: light cleanup of the ironic-integration page
    
    Remove the commands required to configure the nodes, the resources
    and the flavors and refer to the Ironic documentation instead.
    
    This change also removes the only reference in Sahara to the
    old pxe_ipmitool driver, which is going to be removed:
    http://lists.openstack.org/pipermail/openstack-dev/2018-March/128438.html
    
    Remove also the reference to a bug which was fixed for a (long) while.
    
    Change-Id: I5366298f82afb928b7c7520e2eb815d3ff597885
    
  - doc: external link helper for other projects' doc
    
    The external link helper makes sure that the generated link
    respects the current branch (latest for master, <release_name>
    when built on stable/<release_name>.
    
    Do not use it for sahara-tests (which is branchless).
    
    Change-Id: Ie37fe37858a41f54558642ea93a75365668ef573
    
  - Update the command to change the hostname
    
    The instance hostname configuration is temporary, it will fail
    after resart the instance.
    
    Story: 1742369
    Task: 8385
    
    Change-Id: Ibd5891fb2ba7dc825b42a79b7df14d964ee34b1a
    
  - fix tox python3 overrides
    
    We want to default to running all tox environments under python 3, so
    set the basepython value in each environment.
    
    We do not want to specify a minor version number, because we do not
    want to have to update the file every time we upgrade python.
    
    We do not want to set the override once in testenv, because that
    breaks the more specific versions used in default environments like
    py35 and py36.
    
    Change-Id: I03574b12fc9ef07ce459e81c9b8219a3b10bd4da
    Signed-off-by: Doug Hellmann <doug@doughellmann.com>
    
  - Check node processes earlier
    
    When creating a cluster, validate node processes as early as we can.
    
    (The config-recommending step may have some impact on validity, so we
    must leave that as the first call.)
    
    Note that there has been a desire mentioned to move the node process
    validation to occur during cluster template creation. Unfortunately the
    amount of refactoring needed makes the task too daunting to be
    completed now.
    
    Change-Id: Ib5c91e062b32a83268d178417cbc5120d4c57934
    
  - [APIv2]Consolidate cluster creation endpoints
    
    Creation of a single cluster and creation of multiple clusters will now
    share an API endpoint (in the APIv2 case). More specifically, the
    original single-cluster endpoint will accept a `count` parameter in the
    request and the multiple-cluster endpoint has been removed.
    
    We can make this kind of change because APIv2 is still experimental.
    
    Also, when creating multiple clusters, the response will now contain
    all details about the clusters; previously, the response simply gave
    cluster IDs.
    
    Change-Id: I90faf4956a8ea4b4ae31a29382732771fdfddecb
    Story: 2002099
    Task: 19777
    
  - Add support to deploy hadoop 2.7.5
    
    Add hadoop 2.7.5 deployment script into vanilla plugin.
    
    Change-Id: I8f3b4a447d8b76e5a1e3f88e1e2c7f009b433bb6
    
  - Merge "Fix the installation of Swift Hadoop connector (Ambari)"
  - Merge "Restore Ambari with newer JDK security policies"
  - Merge "Switch from sahara-file to tarballs.o.o for artifacts"
  - Restore Ambari with newer JDK security policies
    
    Recent changes in JDK security policies disabled TLSv1, which is used
    by default in the communications between ambari-agent and ambari-server.
    More details here:
    https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.html
    
    In order to restore the functionalities, two changes are needed:
    - Ambari 2.4.3.0, a minor update in the 2.4.x line;
    - a change in the ambari-agent configuration file to force a newer TLS.
    
    Story: 2002012
    Task: 19651
    Change-Id: I3782ce9acb8c895e4e1f3fb9046b54f2a57acdbf
    
  - Fixing java version for Ambari
    
    From SIE java version installed for ambari is 1.8.0, we had the wrong
    one here.
    
    Story: #2002003
    Change-Id: I78da2fb94b0e8a362c7b4daae166849974682960
    
  - Switch from sahara-file to tarballs.o.o for artifacts
    
    Clean the few remaining links to artifacts that still uses sahara-files,
    but they have been available on tarballs.openstack.org for a good while.
    
    Story: 2001996
    Task: 19627
    Change-Id: Ie17550afc7216bce88af5e6164b74420db0040c0
    
  - Deploy using wsgi by default
    
    Complete the switch to wsgi and only use it from now on.
    The default won't be changed for old branches (i.e. this change
    should not be backported).
    
    Story: 2001991
    Task: 19622
    Change-Id: Ia23083d0ec7c17a84e0bb4bc5db970cbfe57882a
    
  - Fix: really install extjs in CDH images at build time
    
    Use the same logic as the configure_extjs snipped used for mapr
    (we should have a common file for all of them, with just different
    parameters).
    It is worth noting that the file is downloaded and copied at runtime
    if not found, but better support the offline case.
    
    Story: 2001992
    Task: 19618
    Change-Id: I5bdded236abd218891cda1bf7d3f25db90c77fdf
    
  - doc: add the redirect for a file recently renamed
    
    Change-Id: I6988795e477fda23a393710e8b455210a3a6ad45
    
  - Merge "Adding Ambari missing versions"
  - Merge "Fix: always use kafka 2.2 for CDH 5.11"
  - Merge "Fix the detection of scala version (now https)"
  - Fix the detection of scala version (now https)
    
    The www.scala-lang.org switched to https and the curl call did not
    account for this scenario, so it failed.
    Switch the address to https and as extra measure add also the -L
    argument, which forces curl to follow the redirects.
    
    Switch also the download URL to https.
    
    Story: 2001964
    Task: 15088
    Change-Id: I9d28c2ad56292998a6b7aaea421b98e136fbf6e1
    
  - Fix the installation of Swift Hadoop connector (Ambari)
    
    The default value of the URL lacks the file name.
    
    Also, fix another wrong URL which should be used as fallback,
    even if it is in fact never used thanks to the default value.
    
    Story: 2001961
    Task: 15083
    Change-Id: I73ec63e65313cf8764197f9e98f538e66d7f629b
    
  - Fix the installation of the Swift Hadoop connector (CDH)
    
    The jar file with the same name already exists because
    it is installed with the rest of the Cloudera packages,
    but it must be overwritten by our version with Swift support.
    So do not exit if the jar already exists.
    
    Story: 2001909
    Task: 14422
    Change-Id: Ibc0a5f7348365f2a6c5658846ee74aa92e78314b
    
  - Merge "fix a typo: s/avaliable/available"
  - fix a typo: s/avaliable/available
    
    Change-Id: I6f9da5c48375ce392bdc71e48bfcfff36677ddb7
    
  - Remove the (now obsolete) pip-missing-reqs tox target
    
    The test does not work anymore with pip 10. For more details, see:
    http://lists.openstack.org/pipermail/openstack-dev/2018-April/130027.html
    
    (I suspect that the development that happened later in OpenStack testing
    made it obsolete anyway).
    
    Change-Id: I2227e70d23e5e7e99a85834ccc7d6ec8a09bf4ae
    
  - Merge "Fix the openstack endpoint create failed"
  - Replace Chinese punctuation with English punctuation
    
    Curly quotes(Chinese punctuation) usually input from Chinese input method.
    When read from english context, it makes some confusion.
    
    Change-Id: I55f81ca701c9e25d499b702aa4f8ab9e09fa2699
    
  - Fix the openstack endpoint create failed
    
    Change-Id: Ie9b95c688dfacd118a1eebed587b015065866210
    
  - Fix: always use kafka 2.2 for CDH 5.11
    
    This is the case in sahara-image-elements already, and the ubuntu
    code path already uses that version.
    
    Story: 2001893
    Task: 14377
    Change-Id: If9364d67f9e1f2af119a438d707ff205be454423
    
  - Adding Ambari missing versions
    
    With the new validation system, image versions on get_image_arguments()
    has to contain all available versions in order to allow cluster creation
    and validation with them all.
    
    Story: #2001888
    Change-Id: I88c2a553512a797099a4b2fbd9e9d204475e755b
    
  - Merge "Adding ntpdate and Scala to mapr image"
  - Merge "Fix MapR dependency on mysql on RHEL"
  - Merge "Extend config-grabbing magic to new oslo.config"
  - Extend config-grabbing magic to new oslo.config
    
    Some changes (in what are admittedly private methods) from oslo.config
    5.x to 6.x broke our hacky bits. These bits have now been adjusted to
    be compatible with a wider range of oslo.config versions.
    
    This delicate code must be maintained in order to support grabbing
    config values from [keystone_authtoken], a behavior which is usually
    discouraged.
    
    Change-Id: I9aaa4a3e9052a61269bb7ffcc642383ad6c5a0d8
    Story: 2001835
    Task: 12598
    
  - Adding ntpdate and Scala to mapr image
    
    Adding missing packages to MapR
    
    Change-Id: I01cba8f8518a334ccd91ae07d9a210c93ac3649b
    Story: #2001833
    Story: #2001834
    
  - Merge "Change doc registering-image image message"
  - Merge "uncap eventlet"
  - Change doc registering-image image message
    
    The Ubuntu message "12,14", The user didn't the
    understand ubuntu message "12,14", The version is
    "12,14" or "12.04" or "12" and "14" both support.
    
    Change-Id: Ic6f3ba069df5e9eff0e434195cab880a4ca9977d
    Signed-off-by: Yuanbin.Chen <cybing4@gmail.com>
    
  - Merge "Support of HDP 2.6"
  - Merge "Remove step upload package to oozie/sharelib"
  - Remove step upload package to oozie/sharelib
    
    It is not robust to upload a special package in a common method.
    However, because of change from Hadoop(2.8.2 and later)side,
    we do need the dependency package "commons-httpclient" in
    oozie/sharelib to let oozie work.
    This patch removes the uploading step in sahara because there is
    a patch 2eedcec728b4f93e4cb226c5137159dcb3ddbfa3 which add
    "commons-httpclient" as a dependency to oozie/sharelib.
    So when we build the oozie binary package, "commons-httpclient"
    will be as a part of it.
    
    Change-Id: I4f20a356efc2dcc51aac2f10105be6aa351f87e8
    
  - uncap eventlet
    
    We will manage the eventlet version using constraints now. See the
    thread starting at
    http://lists.openstack.org/pipermail/openstack-dev/2018-April/129096.html
    for more details.
    
    Change-Id: I6f4802f77fc3b8b1e29321b692e70ccc035f6a21
    Signed-off-by: Doug Hellmann <doug@doughellmann.com>
    
  - Fix MapR dependency on mysql on RHEL
    
    MapR is missing mysql-java-connector and that makes it necessary to have
    subscription enable on RHEL7
    
    Change-Id: I1866ee0e21edd46773e67cb24654f00aadf8e39a
    Story: #2001773
    
  - Merge "Follow the new PTI for document build"
  - correct lower-constraints
    
    Fix the incorrect lower constraints so they match the requirements
    lists.
    
    Change-Id: If55b7438c95b0364d93829392a59439e553722b9
    Signed-off-by: Doug Hellmann <doug@doughellmann.com>
    
  - Support of HDP 2.6
    
    Change-Id: Id770ced1626f0e0e593930267f41aeec83311d0f
    
  - Merge "Preload soci-mysql and soci on RHEL7 images"
  - Follow the new PTI for document build
    
    For compliance with the Project Testing Interface as described in:
    https://governance.openstack.org/tc/reference/project-testing-interface.html
    http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
    
    Remove the '[build_sphinx]' section as described in:
    http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html
    
    Change-Id: Ia8e8d6b4e23c2737a9e948e7c4b682dca697f6a4
    
  - Merge "File copy timesout when file is too big"
  - Merge "add lower-constraints job"
  - Updated from global requirements
    
    Change-Id: Ifcd81e002e4bfe9dbd313040ab285d717b0ced9e
    
  - add lower-constraints job
    
    Create a tox environment for running the unit tests against the lower
    bounds of the dependencies.
    
    Create a lower-constraints.txt to be used to enforce the lower bounds
    in those tests.
    
    Add openstack-tox-lower-constraints job to the zuul configuration.
    
    See http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
    for more details.
    
    Change-Id: I3a1d395b43e77010bf44ba71f5bcf146c01e0098
    Depends-On: https://review.openstack.org/555034
    Signed-off-by: Doug Hellmann <doug@doughellmann.com>
    
  - File copy timesout when file is too big
    
    File copy times out when file is too big due to an problem with paramiko
    write implementation.
    
    The fix proposed comes in two parts:
     1) Changing paramiko file write to putfo;
     2) Increase the default copy file timeout.
    
    Change-Id: I9e9d2873d95923cbd8c4729b3a674dfb1b8c2ec1
    Story: #1705762
    
  - Preload soci-mysql and soci on RHEL7 images
    
    Change-Id: I789e93349b89bf96bddbb5e745cb859c975c4d80
    Story: #1754313
    Task: #8682
    
  - Migration to Storyboard
    
    Sahara projects migrated to storyboard.openstack.org.
    Replace the references to Launchpad, including the rename of
    a documentation page focuse on launchpad (which would
    probably require a redirect on the openstack.org site),
    and clean up some old details about blueprints.
    Fix the bug links in the openstackdocstheme configuration.
    
    Change-Id: I30642356f1b1076a874f14fc43fad234fa9a098d
    
  - Merge "Adding support for RHEL images"
  - Updated from global requirements
    
    Change-Id: I79788e84c73f141fe1f52df27c42e6565f98a86d
    
  - Updated from global requirements
    
    Change-Id: I4d155bdeb7e9f87610c0cfebcdc1aa9b8bc8f818
    
  - Updated from global requirements
    
    Change-Id: I1c428b449f21e7f9b33c10621ff16ca3d3c2f9ba
    
  - Adding support for RHEL images
    
    Adding support for rhel images on the image generation system.
    
    Note.: It is necessary to previous to start packing the image for the
    user to register the image using virt-costumize and also enable the
    necessary REPOS.
    
    Change-Id: Ia8c483d34a26ba0ccfe25b5496cc03af4c1b7808
    
  - Remove unused module
    
    Change-Id: I302aefcb5cac50ed08498922c417278656dbdbdd
    
  - change python-libguestfs to python-guestfs for ubuntu
    
    python-libguestfs is for centos,
    python-guestfs is for ubuntu.
    
    Change-Id: I39b18befbc480fedba4da06ea7ce33375f3358bf
    
  - Updated from global requirements
    
    Change-Id: I49f5ac61be59995f4d2645bef8df51a890e61dad
    
  - Merge "Updated from global requirements"
  - Merge "Imported Translations from Zanata"
  - Merge "Fix Spark EDP job failed in vanilla 2.8.2"
  - Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: Ia6584c2346851cc9e658215f1055912518e07e1d
    
  - Updated from global requirements
    
    Change-Id: Ibce27ba11d6497a2de2ad49b1bd01c5109ce9c7d
    
  - Update mysql connection in configuration-guide.rst
    
    Change-Id: Ibe9ea2d0532d3fc3936d62ae9dd339b701c4d1eb
    
  - Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: Ifdff907cf5adc096d46b5e5539d57b25fafce869
    
  - Merge "Fix documents title format error"
  - Fix Spark EDP job failed in vanilla 2.8.2
    
    Vanilla should specify corresponded hadoop-openstack package
    according to plugin version in spark configuration.
    
    In Vanilla 2.8.2, there is an error that hadoop-openstack
    version was specified to 2.7.1. This causes spark job failed
    in vanilla 2.8.2 because of "No such file".
    
    Change-Id: I5b54d69def7b457715ed60da3663a0153fe94be8
    
  - Fix documents title format error
    
    Change-Id: I9121ec8adc694d85a77116ad18b752cd01b57591
    
  - Migrate the artifact link to sahara-extra, use https
    
    The stable link for artifacts is going to be tarballs.openstack.org/sahara-extra,
    which matches the name of the repository and it is more consistent
    with the way the publishing system works.
    
    Also use https.
    
    Change-Id: Iacce6e6397b2d8d812964d56a33f9ea82901a3d6
    
  - Merge "Change some parameters to be required in api-ref"
  - Updated from global requirements
    
    Change-Id: Ib4f51660b0a6eb700d4e0fa4c3775ccab12bd8d1
    
  - Merge "Adding Ambari 2.4.2.0 to image gen"
  - Merge "Native Zuul v3 jobs (almost all of them)"
  - Updated from global requirements
    
    Change-Id: I64b289f95f53e5c1ea14d383a6b31cf98d789756
    
  - Adding Ambari 2.4.2.0 to image gen
    
    We missed ambari 2.4.2.0 on ambari image gen
    
    Also we are disabling CA checking for Centos/RHEL because the default SSL
    certificate, which is generated when the Ambari server is installed, is
    invalid.
    
    Partial-bug: #1748507
    Change-Id: I272dbab4458c902af404a6365a8a43d56e4ed94e
    
  - Native Zuul v3 jobs (almost all of them)
    
    It defines buildimages/CLI jobs, and reuses the global pylint job
    and few jobs from sahara-tests (scenario, tempest), mimicking
    the existing definitions of the legacy jobs.
    
    Still missing: grenade (there is no common grenade job yet at this
    point).
    
    Change-Id: Ibb57e216410afeef4d55d8ba2576aaacfa2f8c1a
    
  - Change some parameters to be required in api-ref
    
    These three parameters are used in path. Change them to be required.
    
    Change-Id: I9c247b7ebefce59d499c495601b33a3aba7063c0
    
  - Fix the parameter in api-ref
    
    Here, the mapping of parameter 'image_id' is wrong since it is used in path.
    So, fix it.
    
    Change-Id: I582f4bbd39407c27c0875db11d8f70f82bc2576e
    
  - Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I5ac635bd5c446dbfcf280bf173a9b84697184ea0
    
  - Update reno for stable/queens
    
    Change-Id: Ibb813e96d2929ae90cec93a3cb9adc081382bc4c
    
  - Merge "Small doc fixes found during doc day"
  - Small doc fixes found during doc day
    
    As a pre-PTG preparation we did a small walktrough on the Sahara
    documentation. Theses fixes are trivial fixes, mostly making
    documentation consider bare metal and VMs.
    
    Change-Id: I9dcf96700392053cc1ac239793c793f0e4a22dd8
    
  - Fixes for the dashboard guide (title, formatting)
    
    - more specific title for the page;
    - better formatting of the shell commands;
    - break a long line.
    
    Change-Id: I3dfa1c1c8214899adaee2eb0a4ae115c4975d0a8
    
  - Merge "Adding Storm doc"
  - Adding Storm doc
    
    Finally writting this long due documentation for the Storm plugin.
    
    Change-Id: I94bb588a41e181ad3c27d371cfe55938fe0579f7
    Closes-bug: #1696991
    
  - Merge "Switch sahara swift to work with keystone v3"
  - Merge "EDP doc: de-emphasize job binary internals (not in v2)"
  - Switch sahara swift to work with keystone v3
    
    Keystone v2 is gone and we need to update sahara swift to be able to use
    keystone v3.
    
    Change-Id: I65a2495b8afe2bc30a0db192e23c93cd6b71e437
    
  - Replace chinese quotes
    
    Change-Id: I39f1b5efad55a3c20d09bde8280b95c5f146b490
    
  - EDP doc: de-emphasize job binary internals (not in v2)
    
    * Adding deprecation warning about job binary internals in the
      dashboard user guide
    * Changing how deprecation warning is given at userdoc edp page
    * Changing dashboard user guid job examples, now it uses swift
      as type storage for job binaries instead of internals
    
    Change-Id: Ib1083ee6f6e204248e10f17b5ce58fd424e74106
    Partial-Implements: bp remove-job-binary-internal
    
  - Merge "Enable hacking-extensions H204, H205"
  - Enable hacking-extensions H204, H205
    
    This patch enable extensions:
    -[H204] Use assert(Not)Equal to check for equality.
    -[H205] Use assert(Greater|Less)(Equal) for comparison.
    
    Change-Id: If1b09426deba024ce75ee0d2d687c2501a1e141d
    
  - Adding sahara-policy-generator.conf
    
    This conf file is necessary to generate policy file.
    
    Change-Id: I076f897115acdae820330164c7fcaac7b56d3aa4
    Closes-bug: #1745285
    
  - use . instead of source.
    
    Code conventions: Use “.” to source script files
    When you have to source a script file, for example ,
    a credentials file to gain access to user-only or
     admin-only CLI commands, use . instead of source.
    See more:
    http://docs.openstack.org/contributor-guide/writing-style/code-conventions
    
    Change-Id: Icaf20628e7b7480ee4ca4c85f9c5a3802b817070
    
  - Merge "Fix Flask error_handler_spec"
  - Merge "Various server-side fixes to APIv2"
  - Merge "Dynamically add python version into launch_command"
  - Merge "Add support to deploy Hadoop 2.8.2"
  - Merge "Tweak Sahara to make version discovery easier"
  - Merge "Force deletion of clusters"
  - Add support to deploy Hadoop 2.8.2
    
    Add hadoop 2.8.2 deployment script into vanilla plugin.
    
    Change-Id: I6a3f3c035d97bed6ed3dcb99c2feb78f62336817
    
  - Tweak Sahara to make version discovery easier
    
    Or at least, tweak Sahara to be more accomodating to keystoneauth's way
    of doing version discovery.
    
    * Don't require auth to do version discovery
    * Make project ID in URL optional for APIv1 (and also for APIv2, but
      don't go around advertising that... *wink*)
    
    Change-Id: Idb6f734aee26cab5bd629963a66ba01c92760864
    Closes-Bug: #1744350
    
  - Various server-side fixes to APIv2
    
    * Check if plugin_version key is in dict before doing something to it
    * Fix references to non-existent policies
    * Generally, correct how some responses get tweaked (in many cases the
      old way was throwing an error)
    * Fix an incorrect schema
    
    Change-Id: I6b4802a614e4b58678343f12856dd531827dc7b2
    
  - Fix Flask error_handler_spec
    
    The error_handler_spec is supposed to be a dict of dict of dicts, but
    we had it as a dict of dict of functions, for unknown (historical?)
    reasons.
    
    Fix that, so that errors which occur in the Sahara API, but are not
    otherwise caught by `sahara.utils.api.Rest.route` get handled correctly
    by Flask instead of leading to some arcane error far down the stack.
    
    Change-Id: I1e9d5f3fa00308baa7eee101c1f3b5a666cae4aa
    Closes-Bug: #1745236
    
  - Dynamically add python version into launch_command
    
    Ubuntu Xenial or later server won't install Python2 anymore by default.
    Sahara should have the ability to dynamically edit the remotely executed
    python script based on what python is available.
    
    Change-Id: Ie0fdd829d1b0ff019329957fbdbbfd150320b8ab
    Closes-Bug: #1739009
    
  - Updated from global requirements
    
    Change-Id: I070ede3c9e3fb4b21155f258822d0b28ad7ec35d
    
  - Merge "Remove use of unsupported TEMPEST_SERVICES variable"
  - Merge "Stop abusing [keystone_authtoken]"
  - Remove use of unsupported TEMPEST_SERVICES variable
    
    TEMPEST_SERVICES global variable is not supported
    by devstack since long back.
    - I380dd20e5ed716a0bdf92aa02c3730359b8136e4
    - I9c24705e494689f09a885eb0a640efd50db33fcf
    
    Service availability of tempest known services will be
    set by devstack with local check.
    - I02be777bf93143d946ccbb8e9eff637bfd1928d4
    
    This commit removes the unused TEMPEST_SERVICES setting
    Related-Bug: #1743688
    
    Change-Id: I74575c6c6796f875bd4b5b36ced1563c46fb2e47
    
  - Merge "Replace assertFalse/assertTrue(a in b)"
  - Replace assertFalse/assertTrue(a in b)
    
    Replace 'assertFalse(a in b)' with 'assertNotIn(a, b)'
    
    Replace 'assertTrue(a in b)' with 'assertIn(a, b)'
    
    [H204] Use assert(Not)Equal to check for equality. Unit test assertions
    tend to give better messages for more specific assertions. As a result,
    assertEqual(...) is preferred over assertTrue(... == ...), and
    assertNotEqual(...) is preferred over assertFalse(... == ...).
    Off by default.
    
    see more https://docs.openstack.org/hacking/latest/user/hacking.html
    
    Trivial fix.
    
    Change-Id: Ic21fa755faf218bc1c27dc9c839cf6f99f67d287
    
  - Merge "Image generation for MapR"
  - Merge "Update designate manual installation URL"
  - Merge "Update url links in doc files of Sahara"
  - Stop abusing [keystone_authtoken]
    
    That config section should be private for keystonemiddleware, so deprecate and
    discourage its use for trusts creation.
    
    Create a new config section for credentials to create trusts with.
    
    Change-Id: I93b9a9b4c8003463c33439f116b9a72619512b98
    
  - Update url links in doc files of Sahara
    
    replace 'http' with 'https'
    
    Change-Id: Iebee75d464a5f0c4b9b877814ec42ca9be946931
    
  - Updated from global requirements
    
    Change-Id: I6503014ca815329191d08b402d31abee817ae905
    
  - Merge "Changing expected value to job_template_id"
  - Merge "Rename 'SAHARA_AUTO_IP_ALLOCATION_ENABLED' config parameter"
  - Merge "add bugs link in README.rst"
  - Merge "[APIv2]Enable APIv2, experimentally"
  - Changing expected value to job_template_id
    
    Since we are only expecting one job_template_id does not make sense to
    use job_templates_id name here
    
    Change-Id: I2c1425dce3db2b150f798e3c10f7ad5f798226d3
    
  - Updated from global requirements
    
    Change-Id: I7e19da4d367e89fc983cdb9357ed28890e6df8e8
    
  - Updated from global requirements
    
    Change-Id: Ib191bc328f8edd688733e267a9cbb62985f69a3e
    
  - add bugs link in README.rst
    
    add bugs link in README.rst
    
    Change-Id: I8c27fa77a6e19156ebda0f969075762c542e19a2
    
  - Image generation for MapR
    
    Adds image generation and validation for MapR
    
    Change-Id: Ib2d3bf2fa43db96437682d7479df1e897b997674
    
  - Force deletion of clusters
    
    * Add force delete operation to engine using stack abandon
    * Basic unit tests for force delete
    * Necessary removal of "direct engine" code
    * Change schema of APIv2 cluster delete to allow force delete
    * Unit tests for new cluster delete schema
    * Basic docs about force delete
    * Release note
    
    bp sahara-force-delete
    Partial-Bug: #1647411
    
    Change-Id: Ida72677c0a4110cb78edf9d62d8330cd4608ff76
    
  - Rename 'SAHARA_AUTO_IP_ALLOCATION_ENABLED' config parameter
    
    The 'SAHARA_AUTO_IP_ALLOCATION_ENABLED' config parameter in Sahara
    is confusing. If you wish to disable floating IP options during node
    group template creation, set it 'True', but this name sounds like it
    has to do with automatically associating a floating IP. so we can
    rename a meaningful variable name.
    
    Change-Id: Iadb558e9335cac368a340a470807d246a6699d5f
    
  - [APIv2]Enable APIv2, experimentally
    
    Pretty much all the changes from v1->v2 regarding REST semantics,
    payload changes, etc have been completed.
    
    So let's enable APIv2 in an experimental state. It will be needed for
    exposing some new Sahara features like force-delete, decomission of
    a specific node, and hopefully more.
    
    bp v2-api-experimental-impl
    
    Depends-On: I8397ed7c134f0742de8c38466ed3f9035d8103d4
    Depends-On: I102fb2ad16d0256e3a9aa364586332a13826cc90
    Change-Id: I6ee89d52ab6679c34f0a089df4b111796922e171
    
  - Update designate manual installation URL
    
    Change-Id: Id6c1aff0ef2b7299146ea986a3f3628786d1c09d
@hunterirving

This comment has been minimized.

Copy link

@hunterirving hunterirving commented May 30, 2019

I would also like to see support for this new format.

@bitprophet

This comment has been minimized.

Copy link
Member

@bitprophet bitprophet commented May 31, 2019

If anyone else has the spare cycles to check up on the years-old assertions I made early in this ticket, re: whether Cryptography has what we need now, that'd be awesome. I'm optimistic that this is possible to do now, if it hasn't already been done in a fork somewhere (looking at @ploxiln whose fork I do not have time to go over in detail yet).

@bitprophet

This comment has been minimized.

Copy link
Member

@bitprophet bitprophet commented May 31, 2019

Also cc @alex and @reaperhulk as usual on that front ❤️

@bitprophet

This comment has been minimized.

Copy link
Member

@bitprophet bitprophet commented May 31, 2019

Also VERY briefly skimming my old comments here, I'd put this in 3.0's milestone on assumption that this requires a new dependency (python-bcrypt). In the intervening years I've both relaxed my stance on that topic, and again, Cryptography has gone through a lot of development.

I'm going to move this into the p1 milestone with those things in mind - I expect we could do this in Paramiko 2.6 or whatever without making anybody too unhappy, even if it does mean a new dependency is added. Folks grabbing new Paramiko minor releases and not doing it for the new key type and being really mad about some new-to-them dependency, seems like it'd be a very small demographic.

@ploxiln

This comment has been minimized.

Copy link
Contributor

@ploxiln ploxiln commented Jun 1, 2019

the bcrypt dependency was already added in paramiko 2.2.0 for Ed25519 keys - #972

openstack-gerrit pushed a commit to openstack/sahara-tests that referenced this issue Aug 8, 2019
Unfortunately it is not possible to switch to the new, more secure, native
format of OpenSSH >=6.5, because paramiko does not support it:
paramiko/paramiko#602

A similar change has been applied to sahara some time ago:
https://review.opendev.org/605028

Story: 2003674
Task: 35983
Change-Id: I5683245c0a9373e299a647f7f61d3e6a2de284e6
openstack-gerrit added a commit to openstack/openstack that referenced this issue Aug 8, 2019
* Update sahara-tests from branch 'master'
  - Merge "Force the format of ssh key to PEM, at least for now"
  - Force the format of ssh key to PEM, at least for now
    
    Unfortunately it is not possible to switch to the new, more secure, native
    format of OpenSSH >=6.5, because paramiko does not support it:
    paramiko/paramiko#602
    
    A similar change has been applied to sahara some time ago:
    https://review.opendev.org/605028
    
    Story: 2003674
    Task: 35983
    Change-Id: I5683245c0a9373e299a647f7f61d3e6a2de284e6
@bitprophet bitprophet modified the milestones: p1, p0 Nov 25, 2019
bitprophet added a commit that referenced this issue Dec 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.