Merge pull request #531 from /issues/530

fixes #530 - 11.0.0 release
jantman · Apr 21, 2021 · 8921393 · 8921393
2 parents 596c01c + 961f7c8
commit 8921393
Show file tree

Hide file tree

Showing 18 changed files with 283 additions and 86 deletions.
diff --git a/.gitignore b/.gitignore
@@ -65,6 +65,9 @@ include/
 *.iml
 .idea/
 
+# VSCode
+.vscode
+
 dev/.terraform
 dev/iam_policy.json
 dev/terraform.tfstate.backup
@@ -73,3 +76,4 @@ results/
 dev/terraform.tfstate
 man/
 share/
+pyvenv.cfg
diff --git a/.travis.yml b/.travis.yml
@@ -15,15 +15,19 @@ matrix:
       env: TOXENV=py38
       dist: xenial
       sudo: true
-    - python: "3.8"
+    - python: "3.9"
+      env: TOXENV=py39
+      dist: xenial
+      sudo: true
+    - python: "3.9"
       env: TOXENV=docker
       dist: xenial
       sudo: true
       services:
       - docker
-    - python: "3.8"
+    - python: "3.9"
       env: TOXENV=docs
-    - python: "3.8"
+    - python: "3.9"
       env: TOXENV=integration3
 install:
 - git config --global user.email "travisci@jasonantman.com"

diff --git a/CHANGES.rst b/CHANGES.rst
@@ -1,6 +1,35 @@
 Changelog
 =========
 
+.. _changelog.11_0_0:
+
+11.0.0 (2021-04-20)
+-------------------
+
+IMPORTANT - Breaking Changes
+++++++++++++++++++++++++++++
+
+* This release **removes** the EBS ``General Purpose (SSD) volume storage (GiB)`` limit in favor of ``General Purpose (SSD gp2) volume storage (GiB)`` and ``General Purpose (SSD gp3) volume storage (GiB)`` limits, to account for the new gp3 volume type and corresponding AWS service limits.
+* This release **removes** the EBS ``Provisioned IOPS`` and ``Provisioned IOPS (SSD) storage (GiB)`` limits in favor of ``Provisioned IOPS (io1)`` and ``Provisioned IOPS (io2)``, and ``Provisioned IOPS SSD (io1) storage (GiB)`` and ``Provisioned IOPS SSD (io2) storage (GiB)``, respectively, to account for the new ``io2`` EBS volume type and corresponding AWS service limtits.
+
+IMPORTANT - Seeking New Maintainer
+++++++++++++++++++++++++++++++++++
+
+As I commented in `Issue #500 <https://github.com/jantman/awslimitchecker/issues/500>`__, I'm looking for someone to share (and perhaps take over) maintenance of this project. awslimitchecker is, and has always been, a personal-time-only project for me; the only time I've done work on it during my day job is when my employer was experiencing an issue or requested a specific feature. Because of a variety of issues, including changing personal interests and my employer relying on this project much less (following an AWS account restructuring that largely avoids service limits), I've been spending much less time on this project than it deserves. As a result, I'm looking for someone to help with maintenance... at the very least, helping review PRs and get them to a merge-able state. If you're interested, please comment on `Issue #500 <https://github.com/jantman/awslimitchecker/issues/500>`__ or contact me directly. While I am *incredibly* flattered by the offers I've received for sponsorship, paid support, or other financial incentive, I'd ask that anyone who's willing to make that commitment instead dedicate a few hours to working on issues or PRs. I, for my part, will make a concerted effort to quickly merge and release any PRs that meet all of the :ref:`development.pull_request_guidelines`.
+
+All Changes
++++++++++++
+
+* `PR #519 <https://github.com/jantman/awslimitchecker/pull/519>`__ - Add support for ``gp3`` EBS volume type. Thanks to `spockNinja <https://github.com/spockNinja>`__ for this contribution!
+* `Issue #514 <https://github.com/jantman/awslimitchecker/issues/514>`__ / `PR #517 <https://github.com/jantman/awslimitchecker/pull/517>`__ - Fix EKS Fargate profile usage check exception (``KeyError: 'labels'``). Thanks to `jwu2 <https://github.com/jwu2>`__ for first reporting this issue and `sebasrp <https://github.com/sebasrp>`__ for the fix!
+* `Issue #520 <https://github.com/jantman/awslimitchecker/issues/520>`__ / `PR #522 <https://github.com/jantman/awslimitchecker/pull/522>`__ - Add awslimitchecker to `conda-forge <https://github.com/conda-forge/awslimitchecker-feedstock>`__ so it can be installed via `Conda <https://conda.io/>`__. Thanks to `BastianZim <https://github.com/BastianZim>`__ for this.
+* `PR #525 <https://github.com/jantman/awslimitchecker/pull/525>`_ - Fix Prometheus example in docs, thanks to `felixkrohn <https://github.com/felixkrohn>`__.
+* `PR #526 <https://github.com/jantman/awslimitchecker/pull/526>`__ - Fix possible ``KeyError: 'LaunchSpecifications'`` in EC2 service. Thanks to `nitrocode <https://github.com/nitrocode>`__ for this.
+* `PR #528 <https://github.com/jantman/awslimitchecker/pull/528>`_ - Update ECS default limits. Thanks to `arturpriz <https://github.com/arturpriz>`__ for this contribution.
+* `PR #523 <https://github.com/jantman/awslimitchecker/pull/523>`_ - Add support for io2 type EBS volumes. Thanks to `andyjp <https://github.com/andyjp>`__ for this contribution.
+* Add tox and TravisCI support for testing against Python 3.9; bump the ``docker``, ``docs``, and ``integration3`` tox environments from 3.8 to 3.9.
+* Bump the base Docker image from Python 3.8 to 3.9.
+
 .. _changelog.10_0_0:
 
 10.0.0 (2020-12-07)

diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM python:3.8-alpine
+FROM python:3.9-alpine
 
 ARG git_version
 

diff --git a/README.rst b/README.rst
@@ -25,6 +25,10 @@ awslimitchecker
    :alt: gitter.im chat
    :target: https://gitter.im/awslimitchecker/Lobby
 
+.. image:: https://img.shields.io/conda/v/conda-forge/awslimitchecker
+   :alt: Conda
+   :target: https://anaconda.org/conda-forge/awslimitchecker
+
 Master:
 
 .. image:: https://api.travis-ci.com/jantman/awslimitchecker.svg?branch=master
@@ -104,6 +108,8 @@ Requirements
 * Python `VirtualEnv <http://www.virtualenv.org/>`_ and ``pip`` (recommended installation method; your OS/distribution should have packages for these)
 * `boto3 <http://boto3.readthedocs.org/>`_ >= 1.4.6 and its dependency `botocore <https://botocore.readthedocs.io/en/latest/>`_ >= 1.6.0.
 
+Alternatively, awslimitchecker may be installed through `conda <https://docs.conda.io/en/latest/>`_ with ``conda install -c conda-forge awslimitchecker``. This will automatically install the required dependencies as well.
+
 Installation and Usage
 -----------------------
 

diff --git a/awslimitchecker/services/ebs.py b/awslimitchecker/services/ebs.py
@@ -81,9 +81,12 @@ def find_usage(self):
     def _find_usage_ebs(self):
         """calculate usage for all EBS limits and update Limits"""
         vols = 0
-        piops = 0
-        piops_gb = 0
-        gp_gb = 0
+        piops_io1 = 0
+        piops_io1_gb = 0
+        piops_io2 = 0
+        piops_io2_gb = 0
+        gp2_gb = 0
+        gp3_gb = 0
         mag_gb = 0
         st_gb = 0
         sc_gb = 0
@@ -97,10 +100,15 @@ def _find_usage_ebs(self):
         for vol in results['Volumes']:
             vols += 1
             if vol['VolumeType'] == 'io1':
-                piops_gb += vol['Size']
-                piops += vol['Iops']
+                piops_io1_gb += vol['Size']
+                piops_io1 += vol['Iops']
+            elif vol['VolumeType'] == 'io2':
+                piops_io2_gb += vol['Size']
+                piops_io2 += vol['Iops']
             elif vol['VolumeType'] == 'gp2':
-                gp_gb += vol['Size']
+                gp2_gb += vol['Size']
+            elif vol['VolumeType'] == 'gp3':
+                gp3_gb += vol['Size']
             elif vol['VolumeType'] == 'standard':
                 mag_gb += vol['Size']
             elif vol['VolumeType'] == 'st1':
@@ -113,20 +121,36 @@ def _find_usage_ebs(self):
                     " not counting",
                     vol['VolumeType'],
                     vol['VolumeId'])
-        self.limits['Provisioned IOPS']._add_current_usage(
-            piops,
+        self.limits['Provisioned IOPS (io1)']._add_current_usage(
+            piops_io1,
             aws_type='AWS::EC2::Volume'
         )
         self.limits[
-            'Provisioned IOPS (SSD) storage (GiB)'
+            'Provisioned IOPS SSD (io1) storage (GiB)'
         ]._add_current_usage(
-            piops_gb,
+            piops_io1_gb,
+            aws_type='AWS::EC2::Volume'
+        )
+        self.limits['Provisioned IOPS (io2)']._add_current_usage(
+            piops_io2,
+            aws_type='AWS::EC2::Volume'
+        )
+        self.limits[
+            'Provisioned IOPS SSD (io2) storage (GiB)'
+        ]._add_current_usage(
+            piops_io2_gb,
+            aws_type='AWS::EC2::Volume'
+        )
+        self.limits[
+            'General Purpose (SSD gp2) volume storage (GiB)'
+        ]._add_current_usage(
+            gp2_gb,
             aws_type='AWS::EC2::Volume'
         )
         self.limits[
-            'General Purpose (SSD) volume storage (GiB)'
+            'General Purpose (SSD gp3) volume storage (GiB)'
         ]._add_current_usage(
-            gp_gb,
+            gp3_gb,
             aws_type='AWS::EC2::Volume'
         )
         self.limits[
@@ -191,18 +215,18 @@ def _get_limits_ebs(self):
         :rtype: dict
         """
         limits = {}
-        limits['Provisioned IOPS'] = AwsLimit(
-            'Provisioned IOPS',
+        limits['Provisioned IOPS (io1)'] = AwsLimit(
+            'Provisioned IOPS (io1)',
             self,
-            200000,
+            300000,
             self.warning_threshold,
             self.critical_threshold,
             limit_type='AWS::EC2::Volume',
             limit_subtype='io1',
-            quotas_name='Provisioned IOPS'
+            quotas_name='Provisioned IOPS (io1)'
         )
-        limits['Provisioned IOPS (SSD) storage (GiB)'] = AwsLimit(
-            'Provisioned IOPS (SSD) storage (GiB)',
+        limits['Provisioned IOPS SSD (io1) storage (GiB)'] = AwsLimit(
+            'Provisioned IOPS SSD (io1) storage (GiB)',
             self,
             307200,
             self.warning_threshold,
@@ -213,16 +237,51 @@ def _get_limits_ebs(self):
             quotas_unit='GiB',
             quotas_unit_converter=convert_TiB_to_GiB
         )
-        limits['General Purpose (SSD) volume storage (GiB)'] = AwsLimit(
-            'General Purpose (SSD) volume storage (GiB)',
+        limits['Provisioned IOPS (io2)'] = AwsLimit(
+            'Provisioned IOPS (io2)',
+            self,
+            100000,
+            self.warning_threshold,
+            self.critical_threshold,
+            limit_type='AWS::EC2::Volume',
+            limit_subtype='io2',
+            quotas_name='Provisioned IOPS (io2)'
+        )
+        limits['Provisioned IOPS SSD (io2) storage (GiB)'] = AwsLimit(
+            'Provisioned IOPS SSD (io2) storage (GiB)',
+            self,
+            20480,
+            self.warning_threshold,
+            self.critical_threshold,
+            limit_type='AWS::EC2::Volume',
+            limit_subtype='io2',
+            quotas_name='Provisioned IOPS SSD (io2) volume storage',
+            quotas_unit='GiB',
+            quotas_unit_converter=convert_TiB_to_GiB
+        )
+        limits['General Purpose (SSD gp2) volume storage (GiB)'] = AwsLimit(
+            'General Purpose (SSD gp2) volume storage (GiB)',
             self,
             307200,
             self.warning_threshold,
             self.critical_threshold,
             limit_type='AWS::EC2::Volume',
             limit_subtype='gp2',
             ta_limit_name='General Purpose SSD (gp2) volume storage (GiB)',
-            quotas_name='General Purpose (SSD) volume storage',
+            quotas_name='Storage for General Purpose SSD (gp2) volumes',
+            quotas_unit='GiB',
+            quotas_unit_converter=convert_TiB_to_GiB
+        )
+        limits['General Purpose (SSD gp3) volume storage (GiB)'] = AwsLimit(
+            'General Purpose (SSD gp3) volume storage (GiB)',
+            self,
+            307200,
+            self.warning_threshold,
+            self.critical_threshold,
+            limit_type='AWS::EC2::Volume',
+            limit_subtype='gp3',
+            ta_limit_name='General Purpose SSD (gp3) volume storage (GiB)',
+            quotas_name='Storage for General Purpose SSD (gp3) volumes',
             quotas_unit='GiB',
             quotas_unit_converter=convert_TiB_to_GiB
         )

diff --git a/awslimitchecker/services/ec2.py b/awslimitchecker/services/ec2.py
@@ -249,7 +249,7 @@ def _find_usage_spot_fleets(self):
             active_fleets += 1
             cap = fleet['SpotFleetRequestConfig']['TargetCapacity']
             launch_specs = len(
-                fleet['SpotFleetRequestConfig']['LaunchSpecifications'])
+                fleet['SpotFleetRequestConfig'].get('LaunchSpecifications', []))
             total_target_cap += cap
             lim_cap_per_fleet._add_current_usage(
                 cap, resource_id=_id, aws_type='AWS::EC2::SpotFleetRequest')

diff --git a/awslimitchecker/services/ecs.py b/awslimitchecker/services/ecs.py
@@ -187,15 +187,15 @@ def get_limits(self):
         limits['Services per Cluster'] = AwsLimit(
             'Services per Cluster',
             self,
-            2000,
+            5000,
             self.warning_threshold,
             self.critical_threshold,
             limit_type='AWS::ECS::Service'
         )
         limits['Tasks per service'] = AwsLimit(
             'Tasks per service',
             self,
-            2000,
+            5000,
             self.warning_threshold,
             self.critical_threshold,
             limit_type='AWS::ECS::TaskDefinition',
@@ -204,7 +204,7 @@ def get_limits(self):
         limits['Fargate On-Demand resource count'] = AwsLimit(
             'Fargate On-Demand resource count',
             self,
-            500,
+            1000,
             self.warning_threshold,
             self.critical_threshold,
             limit_type='AWS::ECS::TaskDefinition',
@@ -213,7 +213,7 @@ def get_limits(self):
         limits['Fargate Spot resource count'] = AwsLimit(
             'Fargate Spot resource count',
             self,
-            500,
+            1000,
             self.warning_threshold,
             self.critical_threshold,
             limit_type='AWS::ECS::TaskDefinition',

diff --git a/awslimitchecker/services/eks.py b/awslimitchecker/services/eks.py
@@ -139,7 +139,9 @@ def _find_clusters_usage(self):
                     aws_type='AWS::EKS::FargateProfile')
 
                 for selector in profile_selectors:
-                    label_pairs = selector['labels']
+                    label_pairs = selector.get('labels')
+                    if label_pairs is None:
+                        continue
                     self.limits[
                         'Label pairs per Fargate profile selector'
                     ]._add_current_usage(

diff --git a/awslimitchecker/tests/services/result_fixtures.py b/awslimitchecker/tests/services/result_fixtures.py
@@ -141,14 +141,14 @@ class EBS(object):
                 'VolumeType': 'standard',
                 'Iops': None,
             },
-            # 15G general purpose SSD, 45 IOPS
+            # 15G general purpose SSD gp2, 45 IOPS
             {
                 'VolumeId': 'vol-3',
                 'Size': 15,
                 'VolumeType': 'gp2',
                 'Iops': 45,
             },
-            # 30G general purpose SSD, 90 IOPS
+            # 30G general purpose SSD gp2, 90 IOPS
             {
                 'VolumeId': 'vol-4',
                 'Size': 30,
@@ -169,6 +169,20 @@ class EBS(object):
                 'VolumeType': 'io1',
                 'Iops': 300,
             },
+            # 400G PIOPS, 700 IOPS
+            {
+                'VolumeId': 'vol-5',
+                'Size': 400,
+                'VolumeType': 'io2',
+                'Iops': 700,
+            },
+            # 100G PIOPS, 300 IOPS
+            {
+                'VolumeId': 'vol-6',
+                'Size': 100,
+                'VolumeType': 'io2',
+                'Iops': 300,
+            },
             # othertype
             {
                 'VolumeId': 'vol-7',
@@ -188,6 +202,20 @@ class EBS(object):
                 'VolumeType': 'sc1',
                 'Iops': None,
             },
+            # 10G general purpose SSD gp3, 30 IOPS
+            {
+                'VolumeId': 'vol-3',
+                'Size': 10,
+                'VolumeType': 'gp3',
+                'Iops': 30,
+            },
+            # 30G general purpose SSD gp3, 90 IOPS
+            {
+                'VolumeId': 'vol-4',
+                'Size': 30,
+                'VolumeType': 'gp3',
+                'Iops': 90,
+            },
         ]
     }
 
@@ -4618,6 +4646,7 @@ class EKS(object):
             'fargateProfileNames': [
                 'bar',
                 'baz',
+                'profile_no_labels',
             ]
         }
     ]
@@ -4683,5 +4712,15 @@ class EKS(object):
                     },
                 ],
             }
+        },
+        {
+            'fargateProfile': {
+                'fargateProfileName': 'profile_no_labels',
+                'selectors': [
+                    {
+                        'namespace': "test_namespace1",
+                    },
+                ],
+            }
         }
     ]