PyPa

Idioms

Function Simplicity

import logging

def foo(arg):
    """
    Keep it Simple, do one thing and do it well
    """
    if bool(arg):
        # Call and endpoint using arg
    else:
        logging.error(arg)
        raise(arg)

Passing

Use pass in an if/else block as a placeholder when you need to distinguish between a function and a procedure

Linting

Use a minimum set of things to exclude from pylint, for example: #pylint: disable=no-member, invalid-name, line-too-long

Use a comment to setup pylint rules for a single file; use pylintrc to do it for all files

Logging

Log all method locals and method args

String Concat

Use ''.join(foo, bar) instead of foo + bar to distinguish string manipulation from arithmetic and for easier portability to other languages

Pattern Matching

Use polymorphism to implement the multiple dispatch pattern:

class Animal:
    def speak(self):
        pass

class Dog(Animal):
    def speak(self):
        return "Woof!"

class Cat(Animal):
    def speak(self):
        return "Meow!"

def make_animal_speak(animal: Animal):
    return animal.speak()

Use the split and replace methods in tandem multiple times for simple pattern matching. For example:

# Split the connection string into components
# example:
# postgresql://fakeuser:fakepassword@example.com:5432/fakedbname
_, username, password_and_host, port_and_dbname = self.cstring.split(':')
username = username.replace("//","")
password, host = password_and_host.split('@')
port, dbname = port_and_dbname.split('/')
# Copy password to paste during interactive prompt

Use the RegExp module for complex strings and JSONPath(https://pypi.org/project/jsonpath/) and JSONPointer (https://pypi.org/project/jsonpointer/) for everything else to avoid complex if/else/elif or case logic

Boxing

Use autoboxing via the box module(https://pypi.org/project/python-box/) to normalize access to nested key/value pairs

Use getattr to access methods within the box-python lib(https://github.com/cdgriffith/Box/blob/master/test/test_box.py#L856)

Nesting

Rewrite repetitive statements that use nested parentheses as decorators

Rewrite statements that use boolean or logic as addition:

# before
if foo or bar:

# after
if foo + bar == 1

Type Checking

Use multiple dispatch(https://pypi.org/project/multipledispatch/) instead of if/else/elif or case logic to handle method calls which need to handle both structured and unstructured data

Safe Subset

Use a safe subset

State

Use the truths module to encapsulate boolean logic:

Data

Use adapters to maintain API conpatibility layers

Use featuretools to generate mock data

from truths import Truths

# Define your Boolean expressions
expressions = ['(a and b)', 'a and b or x', 'a and (b or x) or d']

# Generate the truth table
my_table = Truths(['a', 'b', 'x', 'd'], expressions)
print(my_table)

Install pip-audit

Add pip-audit to your requirements.txt and run it as part of the CI build to find vulnerabilities in installed modules

Upgrade poetry

Run poetry lock --no-update to update poetry

Clean up Dependencies

• Move devdependencies to dependencies if they are only needed for runtime (boto3, jinja)
• Move dependencies to devdependencies if they are only needed for build time and testing(pytest, urllib3)
• Make sure there is no overlap between dependencies and devdependencies
• Pin dependencies using < if there are compatibility issues

Optimize Deployment

• Modules should be zipped to compress the deployed package
• Make sure lockfiles are in the deployed package

Optimize Configuration

• Make sure variable interpolation is using the correct syntax for XML/JSON/YAML files
• Use functions instead of any other data type if there is an option
• Make sure quotation marks are consistent (double vs single, unix vs windows)
• Use a list instead of a dictionary if the spec requires a list

Use sums of boolean values instead of and/or clauses

If you have conditions based on multiple boolean values, add up the sum of the boolean values rather than using complex logic:

# Replace this:
if a or b:
# With this:
if a + b == 1:

# Replace this:
if a and b:
# With this:
if a + b == 2:

Raise if you do not get the type you expected

Use type to verify assumptions; if they are not true, raise an exception:

    if bool(type(meta) is dict) + bool(type(meta) is list) !== 1
        raise Exception('Wrong')

Raise TypeError or ValueError instead of a generic error

Use NameError, TypeError or ValueError to make exceptions more specific for class, type, or value related code respectively:

class MyException(Exception):
    pass

Use assert to raise errors for impossible calculations

# no matter what, discounted prices cannot be lower than 0 or higher than the listed price
assert 0 <= price <= product['price']

# check if the value of `a plus b` is less than 3
assert a + b < 3, f'No, the answer is {a + b}, which means someone changed the input types from boolean to integer'

# assert a numeric string is numberic
def add_dollar_sign(numeric_string):
    assert numeric_string.isnumeric(), 'Not a numeric string'
    return '$' + numeric_string

Combine logging with uncaught exceptions to trace unexpected errors

def excepthook(*args):
  logging.getLogger().error('Uncaught exception:', exc_info=args)

sys.excepthook = excepthook

assert 1==2, 'Something went wrong'

Use init to share globals

"""
Foo
"""
import os
from foo import Foo
import requests

foo_api_secrets = get_secret("foo")
foo_token = foo_secrets["token"]
foo = Foo(token=foo_token)

from util.foo import foo

Use comments to determine method scope

Add comments first to avoid scope creep

Add a README and link to it in the comments for mission critical details Money

Use Markdown within comments to add complex mixed content, like tables

Use absolute imports instead of relative imports

Instead of:

from . import hubspot

Use the name of the top-level directory instead:

from foo.bar import hubspot

Even if the name of the directory matches the name of the module:

from foo.hubspot import hubspot

Use eval to convert a string to a list

foo = '[[1]]'
bar = eval(foo)
type(bar)

Check how many times a string is found in a list

lst = [1, 2, 3, 'Alice', 'Alice']

indices = [i for i in range(len(lst)) if lst[i]=='Alice']

print(indices)

Convert a one member tuple to a string for use with the DBAPI:

if isinstance(foo, tuple) and len(foo) == 1:
   bar = str(foo).replace(",","")

Convert a multi member tuple to a dict for use with the DBAPI:

   baz = [dict(bar) for bar in foo]

Use return statements instead of assignment statements with the or clause

return {"foo": foo or "N/A"}

vs

try:
   foo = bar
except Exception as e:
   foo = "N/A"
   print e

Use asyncio instead of threads to avoid running out of processes

Use specific exceptions when using except clauses

• Use library specific exceptions instead of the generic Exception class
• Create custom Exception classes when writing user-defined modules
• Use Python's built-in Exception classes when you expect a specific exception

Custom Exception

import argparse

class MyArgumentParser(argparse.ArgumentParser):
    def error(self, message):
        raise argparse.ArgumentError(None, message)

parser = MyArgumentParser(description="Example parser")
parser.add_argument('arg', type=str, help='An argument')

try:
    args = parser.parse_args()
except argparse.ArgumentError as e:
    print(f"Argument error: {e}")

Use nested try/except blocks to swallow errors:

try:
    foo
    try:
        bar
    except Exception as e:
        print("inner")
except Exception as e:
    print("outer")

Use urllib3.exceptions.ConnectionResetError in an except block to implement a retry mechanism

except urllib3.exceptions.ConnectionResetError:
     print(f"ConnectionResetError encountered. Retrying {retries}/{max_retries}...")
     time.sleep(retry_delay)
     if retries < max_retries:
         foo(bar, baz)
     else:
         raise("Max retries reached. Stopping")

Use raise and except to make non-200 responses throw exceptions

Use finally to print logs and timing statistics

    try:
        foo = bar
    except:
        bar = baz
    finally:
        print(foo)
        print(bar)
        print(baz)

Use the following built-ins to distinguish strings:

int(min("0", "hi")) # if you expect a numeric string, but get a alphanum string, this returns 0

str(max("9999", "A")) # if you expect a alphanum string, but get a numeric string, this returns 9999

If a sequence is length zero, it is falsy, so no need to check > 0

search result = []

if foo:
    print(f"foo is empty: {len(foo)}")

If you check the length of a dictionary, it will return the number of keys

   foo = {"hi":"mom","my":"name","is":"kid"}
   print(f"number of keys in foo: {len(foo)}")

Use getattr to check for existence in an object

foo = str(getattr(name, 'first_name', None))

Use a dictionary for caching

foo_cache = {}

if str(first_name) not in name_cache:
   new_name = search_name(first_name)
   if new_name:
       name_cache["first_name"] = new_name

Use .get() and or to search dictionaries

# if get() returns None, the or statement will return a string which can be parsed by the `in` operator
if str(foo) not in (bar.get("baz", "") or ""):
    bar["baz"] = str(foo)

Use split and join rather than string manipulation

number_list = ["1 2 3 4 "].strip().split(' ')
# remove extra whitespace between list indices
normalized_number_list = [index.strip() for index in number_list]
normalized_number_list.remove(4)
# convert list back to a string 
foo = " ".join(normalized_number_list)
# use the new length as a separate variable
foo_length = len(normalized_number_list)

Use str(), int() and min() to defensively add numbers

# Normalize all inputs as a string
foo = str("error")
bar = str(1)
# fallback to the numeric value if casting both as an integer fails
try:
    baz = int(foo) + int(bar) 
except:
    baz = int(min(foo, bar))

Logging to stdout and stderr

import logging
logging.info('This is the existing protocol.')
FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s"
logging.basicConfig(format=FORMAT)
d = {'clientip': '192.168.0.1', 'user': 'fbloggs'}
logging.warning("Protocol problem: %s", "connection reset", extra=d)

Fundamentals

isinstance => instanceOf

min/max => type coercion

repr => valueOf / toString

assert => console.assert

with => context scoping

The with statement makes access to named references inefficient, because the scopes for such access cannot be computed until runtime.

Use strict standards for loops:

• use a break to get out of a loop, especially if it uses recursive calls
• check a loop variable is a list before iterating
• check a loop variable has more than the zero index before iterating; otherwise change the type to a dictionary

Use the following pattern to create a switch/case statement:

from collections import namedtuple

Case = namedtuple('Case', ['condition', 'code'])

cases = (Case('i > 0.5',
            """print 'greater than 0.5'"""),

         Case('i == 5',
            """print 'it is equal to 5'"""),

         Case('i > 5 and i < 6',
            """print 'somewhere between 5 and 6'"""))

def switch(cases, **namespace):
    for case in cases:
        if eval(case.condition, namespace):
            exec(case.code, namespace)
            break
    else:
        print 'default case'

switch(cases, i=5)

Use the following pattern to create a default value:

>>> li1 = None
>>> li2 = [1, 2, 3]

#  li1 is None so li2 is assigned 
a = li1 or li2

Gotchas

If you define a method with a default argument, its mutations will be cached, and all future calls will merge the new data and the mutated data.

Use None as a default argument to avoid this.

If you bind a default value to a lambda, it will be bound to the method:

f = lambda x=x: x

Decorating a Python method with staticmethod ensures that self will not be provided as an argument. Unlike other methods in Python, the first argument is always the class object.

Use the following argument to the print statement to clear the output buffer: flush=true

for i in range(10):
    print(i, end=" ", flush=True)
    time.sleep(.2)
    print()

Use raise to bubble up errors raised in a try clause:

try:
    raise NameError('HiThere')
    except NameError:
        print('An exception flew by!')
        raise

Use if/else and raise inside a try clause to conditionally bubble up non-programmatic errors in a REST API:

try:
   if foo:
       return jsonify({"foo":str(foo)})
   else:
       raise Exception(f'foo is None so the response cannot be parsed')
except Exception as e:
   return jsonify({"exception": str(e)})

Use the build environment variable to return try/except data in responses when for QA/Test environments:

try:
    foo()
except Exception as e:
    if BUILD_ENV == 'develop':
        return {"error": e, status: 500}
    else:
        return {"data": [], status: 500}

Use SQL regexp methods as a first resort, and re as a second resort (JS as a last resort) to do complex string manipulation:

import re

class Solution(object):
    
    def __init__(self):
        self.email_cache = {}
    
    def uniqueLocalName(self, email):
        sanitized_plus = re.sub('\+([^@]+)', '', email)
        unsanitized_local_name, domain_name = re.split('@', sanitized_plus)
        sanitized_local_name = re.sub(r'([^\.]+)\.?', r"\1", unsanitized_local_name)
        sanitized_email = sanitized_local_name + '@' + domain_name
        print(sanitized_email)
                                 
        if sanitized_email not in self.email_cache:
            self.email_cache[sanitized_email] = sanitized_email
            

    def numUniqueEmails(self, emails):
        """
        :type emails: List[str]
        :rtype: int
        """
        for email in emails:
            self.uniqueLocalName(email)
            
        return len(self.email_cache.keys())

Always catch exceptions when creating lists of dictionaries. If an exception happens, assign it to a variable in the except block then return it as a msg key/value pair. Otherwise, use the list index as the msg value:

        try:
            results = None
            payload = []
            results = db_session.execute(query).fetchall()
        except Exception as e:
            print(f'query: {e}')
            results = e
        finally:
            print(f'uuid: {uuid}')

        if isinstance(results, list):
            for idx, row in enumerate(results):
                payload.append(
                    {
                        "id": str(row[0]),
                        "msg": str(idx),
                    }
                )
        else:
                payload.append(
                    {    
                        "id": "",
                        "msg": str(results),
                    }
                )

Use a decorator to create a repeatable method for timing:

import time

def timer(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        print(f"Time taken to run {func.__name__}: {end_time - start_time:.4f} seconds")
        return result
    return wrapper

@timer
def example_method():
    # Simulate a task taking some time
    time.sleep(2)
    print("Method execution complete.")

# Call the method
example_method()

Use an enum class to store constants:

class Day(IntEnum):
    MONDAY = 0
    TUESDAY = 1
    WEDNESDAY = 2
    THURSDAY = 3
    FRIDAY = 4
    SATURDAY = 5
    SUNDAY = 6

Run things locally to get stack traces that are obscured by the logging needle in a haystack problem.

flask dev

Use an event loop to control asyncio:

import asyncio

def hello_world(loop):
    """A callback to print 'Hello World' and stop the event loop"""
    print('Hello World')
    loop.stop()

loop = asyncio.new_event_loop()

# Schedule a call to hello_world()
loop.call_soon(hello_world, loop)

# Blocking call interrupted by loop.stop()
try:
    loop.run_forever()
finally:
    loop.close()

Use type hint to generalize type checking:

from typing import Dict, List, Optional

class Node:
    ...

class SymbolTable(Dict[str, List[Node]]):
    def push(self, name: str, node: Node) -> None:
        self.setdefault(name, []).append(node)

    def pop(self, name: str) -> Node:
        return self[name].pop()

    def lookup(self, name: str) -> Optional[Node]:
        nodes = self.get(name)
        if nodes:
            return nodes[-1]
        return None

'''
SymbolTable is a subclass of dict and a subtype of Dict[str, List[Node]].
'''

Use Slack webhooks as a poor man's Persistent Logger

Marshmallow

from marshmallow import Schema, fields

class RSSItemSchema(Schema):
    title = fields.String()
    link = fields.Url()
    description = fields.String()
    pubDate = fields.DateTime()
    guid = fields.String()

from marshmallow import Schema, fields

class SAMLAssertionSchema(Schema):
    issuer = fields.String()
    subject = fields.String()
    audience = fields.String()
    conditions = fields.String()
    authn_statement = fields.String()
    attribute_statement = fields.String()

Abstract Syntax Tree

import ast
import astunparse

class PrintVisitor(ast.NodeTransformer):
    def visit_Print(self, node):
        # Replace the old print statement with a new print function
        new_node = ast.Expr(
            value=ast.Call(
                func=ast.Name(id='print', ctx=ast.Load()),
                args=node.values,
                keywords=[],
            )
        )
        return ast.copy_location(new_node, node)

def convert_print_statements(source_code):
    # Parse the source code into an AST
    tree = ast.parse(source_code)

    # Transform the AST
    PrintVisitor().visit(tree)

    # Generate the new source code from the AST
    new_source_code = astunparse.unparse(tree)

    return new_source_code

# Read the old source code
with open('old.py', 'r') as f:
    old_source_code = f.read()

# Convert the print statements
new_source_code = convert_print_statements(old_source_code)

# Write the new source code
with open('new.py', 'w') as f:
    f.write(new_source_code)

List to CSV

Wrap each list item in braces: writer.writerow([uuid_str])

Docstrings

Use curly braces to do variable substitution within docstrings

name = "Foo"
bar = f"""
Hi, {name}!
"""
print(bar)

Index out of range

Use a try/catch block, an initial assignment statement, and a nested try/catch block to test and log index access:

try:
     foo = None

        try:
            foo = results["data"]
        except (KeyError, Exception) as e:
            logging.warning(f"failed to get results for: {results}")
        finally:
            foo = foo or {"ok": False}

Open Source Ownership and Auditing

• https://github.com/di
• https://github.com/pypa/pip-audit

Glossary

• https://wiki.python.org/moin/PythonGlossary
• https://wiki.c2.com/?PythonLanguage

Grammar

• https://docs.python.org/3/reference/grammar.html
• https://docs.python.org/3/faq/programming.html
• http://stupidpythonideas.blogspot.com/2016/01/for-each-loops-should-define-new.html

Gotchas

• https://docs.python.org/3/tutorial/errors.html
• https://wiki.c2.com/?PythonProblems
• https://python-docs.readthedocs.io/en/latest/writing/gotchas.html
• http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html
• https://www.hackerfactor.com/blog/index.php?/archives/825-8-Reasons-Python-Sucks.html

Data Model

• https://docs.micropython.org/en/latest/genrst/core_language.html
• https://www.oreilly.com/library/view/fluent-python/9781491946237/ch01.html
• http://www.srikanthtechnologies.com/blog/python/mro.aspx

pip

• https://stackoverflow.com/questions/13270877/how-to-manually-install-a-pypi-module-without-pip-easy-install
• https://stackoverflow.com/questions/5226311/installing-specific-package-versions-with-pip

__init / sys.path

• https://copdips.com/2018/06/import-python-module-with-sys-path-when-without-init-file.html
• https://askubuntu.com/questions/470982/how-to-add-a-python-module-to-syspath
• https://unix.stackexchange.com/questions/10113/configure-python-to-include-another-directory-when-looking-for-packages
• http://wiki.micropython.org/Importing-Modules

command-line args

• https://opensource.com/article/17/3/python-tricks-artists-interactivity-Python-scripts

Executable Scripts as Modules

Separate the __main__ logic for the module itself. For example:

def my_function():
    # Your function implementation here

if __name__ == "__main__":
    # Code to run when the script is executed directly
    print("This will only run if you run the script explicitly, not import it")

Activestate

• https://medium.com/better-programming/how-do-you-split-a-list-into-evenly-sized-chunks-952c4c22e762
• http://code.activestate.com/recipes/425397-split-a-list-into-roughly-equal-sized-pieces/
• http://code.activestate.com/recipes/303279-getting-items-in-batches/
• http://code.activestate.com/recipes/579075-de-chunk-and-decompress-http-body/

Recipes

• http://code.activestate.com/recipes/496716-a-dict-proxy-metaclass/
• https://medium.com/@hrishikesh.pandey9955/how-to-escape-the-circular-import-in-python-5a0f986ddf55
• https://docs.python.org/3/reference/datamodel.html
• https://www.python.org/doc/essays/metaclasses/
• https://en.wikibooks.org/wiki/Python_Programming/Reflection

CMIS

• https://python.hotexamples.com/examples/cmislib.model/CmisClient/-/python-cmisclient-class-examples.html
• https://pypi.org/project/cmislib3/

Reflection

• https://stackoverflow.com/qugamstions/10875442/possible-to-change-a-functions-repr-in-python
• https://stackoverflow.com/questions/1350397/java-equivalent-of-python-repr
• https://chrispenner.ca/posts/python-tail-recursion

Sample Projects

• https://github.com/pallets/flask/blob/master/setup.py
• https://github.com/Delgan/loguru/blob/master/setup.py
• https://github.com/edgewall/trac/blob/trunk/setup.py
• https://elder.dev/posts/open-source-virtual-background/

Sample setup.py

from distutils.core import setup
from setuptools import find_packages

setup(
    name="foobarbaz",
    version="0.9.8",
    description="utility belt",
    author="Foo Bar Bazman",
    author_email="foobarbaz@http://foobarbaz.example.com",
    url="",
    packages=find_packages(),
    package_data={'config': ['README.md']}, # full path: ~/foobarbaz/config/README.md
    install_requires=[
        "hubspot-api-client==3.4.2",
        "python-box>=5.3.0",
        "stripe==2.42.0",
    ],
)

from typing import Dict, List, Optional

class Node:
    ...

class SymbolTable(Dict[str, List[Node]]):
    def push(self, name: str, node: Node) -> None:
        self.setdefault(name, []).append(node)

    def pop(self, name: str) -> Node:
        return self[name].pop()

    def lookup(self, name: str) -> Optional[Node]:
        nodes = self.get(name)
        if nodes:
            return nodes[-1]
        return None

SymbolTable is a subclass of dict and a subtype of Dict[str, List[Node]].

Callable vs Non-Callable Native Properties

my_string = "Hello, World!"
string_methods = [method for method in dir(my_string) if callable(getattr(my_string, method))]
print("String methods:")
for method in string_methods:
    print(method)

Wrapper Method

Use a class to return a method wrapped in your own library rather than a method to avoid returning a function that has to be called rather than a class which includes the method as a property:

import bar

def foo:
    return bar

# import foo
# foo = foo()
# baz = foo.bar(True)

import bar

class Foo:
    def __init__(self):
        self.bar = bar

## import Foo
foo = Foo()
baz = foo.bar(True)

Import Error Handling

import traceback
import importlib

def find_bar_variable(module_path):
    try:
        # Import the module dynamically
        module = importlib.import_module(module_path)

        # Check if 'bar' is a callable attribute (method or function)
        if hasattr(module, 'bar') and callable(getattr(module, 'bar')):
            # Call the 'bar' method and print the returned value
            result = getattr(module, 'bar')()
            print(f"Variable returned by 'bar': {result}")
        else:
            print("Method 'bar' not found in the module.")
    except Exception as e:
        print(f"Exception occurred: {e}")
        traceback.print_exc()
    finally:
        print(f"Call Stack: {traceback.print_stack()}")

# Example usage:
module_path = 'your_module_name'  # Replace with the actual module name or file path
find_bar_variable(module_path)

Class and Method Definitions

• DECOUPLE, Decouple, decouple
• Use Class definitions to organize state, define naming conventions, and enforce the order of operations
• Use arguments to make things testable

Method Tracing

import sys
from functools import wraps

class TraceCalls(object):
    """ Use as a decorator on functions that should be traced. Several
        functions can be decorated - they will all be indented according
        to their call depth.
    """
    def __init__(self, stream=sys.stdout, indent_step=2, show_ret=False):
        self.stream = stream
        self.indent_step = indent_step
        self.show_ret = show_ret

        # This is a class attribute since we want to share the indentation
        # level between different traced functions, in case they call
        # each other.
        TraceCalls.cur_indent = 0

    def __call__(self, fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            indent = ' ' * TraceCalls.cur_indent
            argstr = ', '.join(
                [repr(a) for a in args] +
                ["%s=%s" % (a, repr(b)) for a, b in kwargs.items()])
            self.stream.write('%s%s(%s)\n' % (indent, fn.__name__, argstr))

            TraceCalls.cur_indent += self.indent_step
            ret = fn(*args, **kwargs)
            TraceCalls.cur_indent -= self.indent_step

            if self.show_ret:
                self.stream.write('%s--> %s\n' % (indent, ret))
            return ret
        return wrapper

And here's how we can use it:

@TraceCalls()
def iseven(n):
    return True if n == 0 else isodd(n - 1)

@TraceCalls()
def isodd(n):
    return False if n == 0 else iseven(n - 1)

print(iseven(7))

System Level Exception Handling

import sys

def custom_exception_hook(exctype, value, traceback):
    print(f"Caught {exctype.__name__}: {value}")
    # Handle the exception or perform other actions here

# Set the custom exception hook
sys.excepthook = custom_exception_hook

# Your code goes here...
# Any unhandled exceptions will now be caught by the custom_exception_hook.

metaclasses

Use tuples to return multiple values and implement multiple dispatch based on the method signature of a class method

Creating a metaclass can allow you to add behavior to a class, for example, adding dot notation to a dictionary:

1st party vs 3rd party libraries

• Look for a first party library first
• If there is one, try it
• If not, look for a third party library second
• If there is one, try it
• Otherwise, build one yourself
• Open source it

class DotDict(dict):
    def __getattr__(self, attr):
        return self.get(attr)

    def __setattr__(self, key, value):
        self[key] = value

    def __delattr__(self, item):
        if item in self:
            del self[item]

which can be used like this:

d = DotDict()
d.foo = 'bar'  # equivalent to d['foo'] = 'bar'
print(d.foo)  # equivalent to print(d['foo'])

• https://developer.ibm.com/tutorials/ba-metaprogramming-python/

• https://pypi.org/project/phantom-types/

• https://stackoverflow.com/questions/44799976/python-exception-how-does-the-args-attribute-get-automatically-set

• https://airbrake.io/blog/python-exception-handling/attributeerror

• https://rszalski.github.io/magicmethods/

• https://igraph.org/python/doc/abc.ABCMeta-class.html

• https://bugs.python.org/issue5250

2to3

2to3 is one good example of a good use of eval

• https://github.com/erdc/python/blob/master/Doc/library/2to3.rst
• https://github.com/nvaccess/2to3/tree/master/lib2to3
• https://devtut.github.io/python/dynamic-code-execution-with-exec-and-eval.html#evaluating-a-string-containing-a-python-literal-with-ast-literal-eval
• https://stackoverflow.com/questions/12168978/evalinput-in-python-2to3

Installation/Upgrading cpython

• https://computingforgeeks.com/how-to-install-python-on-debian-linux/
• https://linuxconfig.org/how-to-change-from-default-to-alternative-python-version-on-debian-linux
• https://www.activestate.com/blog/maintain-or-migrate-the-python-2-migration-conundrum-in-2022/
• https://www.activestate.com/blog/maintain-or-migrate-the-python-2-migration-conundrum-in-2022/

pylint

• http://pylint-messages.wikidot.com/messages:e1136
• https://github.com/PyCQA/pylint/issues/289
• http://pylint.pycqa.org/en/latest/technical_reference/features.html#pylint-checkers-options-and-switches
• https://en.wikipedia.org/wiki/Python_syntax_and_semantics
• https://wiki.python.org/moin/Why%20is%20Python%20a%20dynamic%20language%20and%20also%20a%20strongly%20typed%20language
• https://upload.wikimedia.org/wikipedia/commons/9/91/Python_Programming.pdf

pytest

• https://docs.pytest.org/en/7.1.x/example/simple.html
• https://docs.pytest.org/en/7.1.x/how-to/assert.html
• https://the-examples-book.com/programming-languages/python/pytest
• https://github.com/pytest-dev/pytest/blob/master/testing/test_monkeypatch.py#L49
• https://docs.pytest.org/en/5.4.3/monkeypatch.html?highlight=patch
• http://docs.activestate.com/activepython/3.5/pkg/pytest/contents.html
• https://www.patricksoftwareblog.com/monkeypatching-with-pytest/
• https://gist.github.com/alexmic/7482313
• https://gist.github.com/mariocesar/11167b45eb2fd163f6dc0ac341e61de7
• https://github.com/serverstf/python-valve/blob/master/tests/test_api.py
• https://www.linuxjournal.com/content/testing-your-code-pythons-pytest-part-ii
• https://semaphoreci.com/community/tutorials/mocks-and-monkeypatching-in-python

unittest lib

• https://github.com/microformats/xoxo/blob/master/testxoxo.py
• https://gist.github.com/jaysonrowe/4057289

Mock Servers

• https://github.com/alm0ra/mockafka-py
• https://s3.amazonaws.com/assets.datacamp.com/production/course_15974/slides/chapter2.pdf

jupyter

• https://pypi.org/project/nba-api/
• https://pypi.org/project/yahoo-fantasy-api/
• https://twitter.com/py_ball_

numpy

• https://numpy.org/doc/2.0/reference/generated/numpy.genfromtxt.html
• https://cs231n.github.io/python-numpy-tutorial/
• https://www.r-bloggers.com/2012/06/comparing-performance-in-r-foreachdosnow-sas-and-numpy-mkl/

pandas

• https://pbpython.com/pandas_transform.html
• https://docs.sdv.dev/rdt/usage/hypertransformer/transformation#transform
• https://www.justintodata.com/pandas-groupby-with-python/
• https://www.dataquest.io/blog/pandas-big-data/
• https://pythonspeed.com/articles/indexing-pandas-sqlite/
• https://sdsawtelle.github.io/blog/output/large-data-files-pandas-sqlite.html
• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_pickle.html
• https://crypto.stackexchange.com/questions/3917/how-to-avoid-a-chicken-and-egg-scenario-with-encrypting-passwords
• https://pypi.org/project/nba-api/

Example

import os
import pandas as pd

def concatenate_csv_files(starting_value, file_path):
    # List all files in the given directory
    all_files = os.listdir(file_path)
    
    # Filter files that start with the specified value and end with .csv
    csv_files = [file for file in all_files if file.startswith(starting_value) and file.endswith('.csv')]
    
    # Sort the list of files by name
    csv_files.sort()

    # List to hold DataFrames
    dataframes = []
    
    # Read each CSV file and append the DataFrame to the list
    for file in csv_files:
        df = pd.read_csv(os.path.join(file_path, file))
        dataframes.append(df)
    
    # Concatenate all DataFrames
    final_df = pd.concat(dataframes, ignore_index=True)
    
    # Output the final DataFrame to a CSV file
    output_file = os.path.join(file_path, 'output.csv')
    final_df.to_csv(output_file, index=False)
    
    print(f"Concatenated {len(csv_files)} files into {output_file}")

# Example usage
concatenate_csv_files('data_', '/path/to/your/csv/files')

arguments

• https://linux.die.net/diveintopython/html/power_of_introspection/optional_arguments.html

special methods

• https://www.pythonlikeyoumeanit.com/Module4_OOP/Special_Methods.html

return values

• https://bugs.python.org/issue30220

internal boxing and unboxing

• https://mail.python.org/pipermail/python-dev/2015-March/138669.html

switch/case method dispatch table

• https://www.python.org/dev/peps/pep-3103/
• https://daobook.github.io/peps/pep-0238/
• https://www.pydanny.com/why-doesnt-python-have-switch-case.html
• https://www.journaldev.com/15642/python-switch-case

Performance Profiling

• https://marcobonzanini.com/2015/01/05/my-python-code-is-slow-tips-for-profiling/
• https://wiki.python.org/moin/PythonSpeed/PerformanceTips
• https://wiki.python.org/moin/TimeComplexity

Debugging Modules

• https://docs.pytest.org/en/latest/pythonpath.html
• https://stackoverflow.com/questions/37233140/python-module-not-found
• http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html
• https://help.pythonanywhere.com/pages/DebuggingImportError/
• https://nerdparadise.com/programming/python/import2vs3

Recipes

• https://code.activestate.com/recipes/580616-python-method-chaining-examples/
• https://dbader.org/blog/records-structs-and-data-transfer-objects-in-python
• https://pypi.org/project/dokuwiki/

site-packages

• https://stackoverflow.com/questions/14295680/unable-to-import-a-module-that-is-definitely-installed
• https://leemendelowitz.github.io/blog/how-does-python-find-packages.html
• https://docs.python.org/3/library/site.html

String Formatting

• https://stackoverflow.com/questions/3542714/variable-interpolation-in-python
• https://riptutorial.com/python/example/1077/str-format-and-f-strings--format-values-into-a-string
• https://docs.python.org/3/library/string.html

Async

• https://peps.python.org/pep-3148/
• https://wiki.python.org/moin/Generators
• https://www.trek10.com/blog/aws-lambda-python-asyncio/
• https://stackoverflow.com/questions/39038358/function-chaining-in-python

Decorators / Mixins

• https://peps.python.org/pep-3107/
• https://effectivepython.com/2019/12/18/prefer-class-decorators-over-metaclasses
• https://flask-login.readthedocs.io/en/latest/_modules/flask_login/mixins.html
• https://www.residentmar.io/2019/07/07/python-mixins.html
• https://wiki.python.org/moin/PythonDecoratorLibrary
• https://pypi.org/project/fastapi-shell/

Patterns

• https://google.github.io/styleguide/pyguide.html
• https://www.tutorialspoint.com/python_design_patterns/index.htm
• https://peps.python.org/pep-0484/
• https://pymotw.com/2/abc/

Parsers

• https://peps.python.org/pep-0305/
• https://pypi.org/project/yamlpath/
• https://pypi.org/project/jsonpath/
• https://pypi.org/project/toml-cli/
• PythonAnywhere

Quiz

• https://www.vskills.in/practice/python-oop-test
• https://www.tutorialspoint.com/python3/python_online_quiz.htm
• https://www.tutorialspoint.com/python/python_online_test.htm
• https://www.w3schools.com/python/pandas/pandas_quiz.asp
• https://www.w3schools.com/python/python_quiz.asp
• https://pbpython.com/markdown-email.html

Multiprocessing

• https://data-flair.training/blogs/python-multiprocessing/
• https://www.blog.pythonlibrary.org/2016/08/02/python-201-a-multiprocessing-tutorial/
• https://www.python.org/dev/peps/pep-0371/

Lazy Eval

• https://stackoverflow.com/questions/21377020/python-how-to-do-lazy-debug-logging/22204021
• https://pypi.org/project/elk/
• https://news.ycombinator.com/item?id=19991634

JWT

• https://github.com/jpadilla/pyjwt/issues?q=is%3Aissue+bearer+
• https://stackoverflow.com/questions/37975523/pyjwt-raises-an-error-on-decode

WSGI/ASGI

• https://www.python.org/dev/peps/pep-3333/#the-start-response-callable
• https://www.mediawiki.org/wiki/Toolserver:Python_WSGI
• https://testdriven.io/blog/fastapi-crud/
• https://asgi.readthedocs.io/en/latest/introduction.html
• https://buildmedia.readthedocs.org/media/pdf/trac/latest/trac.pdf
• https://programming.vip/docs/exception-handling-of-flask-development-skills.html

SSI

• https://modpython.org/live/current/doc-html/ssi.html
• https://stackoverflow.com/questions/31767482/

Sample Code

• https://mkyong.com/python/
• https://www.askpython.com/python
• https://runestone.academy/runestone/books/published/thinkcspy/Lists/TupleAssignment.html
• https://6191.mit.edu/_static/spring23/resources/references/minispec_sequential.pdf

References

• https://svn.python.org/projects/python/trunk/Lib/
• https://urllib3.readthedocs.io/en/stable/reference/urllib3.exceptions.html
• https://en.wikipedia.org/wiki/Standard_streams#Standard_output_(stdout)
• https://dev.to/rf_schubert/how-to-create-a-pip-package-and-host-on-private-github-repo-58pa
• https://www.linode.com/docs/applications/project-management/how-to-create-a-private-python-package-repository/
• https://www.poftut.com/how-to-update-upgrade-a-python-package-with-pip/
• https://www.tutorialdocs.com/tutorial/pip/quickstart.html
• https://medium.com/@thucnc/how-to-publish-your-own-python-package-to-pypi-4318868210f9
• https://pip.readthedocs.io/en/1.4.1/usage.html
• https://betterscientificsoftware.github.io/python-for-hpc/tutorials/python-pypi-packaging/
• https://opensource.com/sites/default/files/gated-content/cheat_sheet_pip.pdf
• https://docs.pytest.org/en/latest/goodpractices.html
• https://www.eventbrite.com/engineering/packaging-and-releasing-private-python-code-pt-2/
• https://blog.ionelmc.ro/2014/06/25/python-packaging-pitfalls/
• https://www.guidodiepen.nl/2019/02/implementing-a-simple-plugin-framework-in-python/
• https://www.python.org/dev/peps/pep-0420/
• https://codereview.stackexchange.com/questions/70268/list-all-classes-in-a-package-directory/70282
• https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html
• https://pymotw.com/2/pkgutil/
• https://tomassetti.me/python-reflection-how-to-list-modules-and-inspect-functions/
• https://www.programcreek.com/python/example/5302/pkgutil.walk_*
• https://www.tutorialspoint.com/package-extension-utility-in-python
• https://timothybramlett.com/How_to_create_a_Python_Package_with___init__py.html
• https://blog.quiltdata.com/import-almost-anything-in-python-an-intro-to-module-loaders-and-finders-f5e7b15cda47
• https://flask.palletsprojects.com/en/1.1.x/patterns/packages/
• https://stackoverflow.com/questions/60161503/python-importlib-loader-exec-module-to-run-a-function-of-a-py-file
• https://pymotw.com/3/importlib/index.html
• https://itnext.io/python-essentials-for-node-js-developers-708bb9487d70
• https://www.w3schools.com/python/python_ref_glossary.asp
• https://riptutorial.com/Download/python-language.pdf
• https://stackoverflow.com/users/20862/ignacio-vazquez-abrams

ipinfo API Design

use Geo::IPinfo;
my $access_token = 'your_api_token_here';
my $ipinfo = Geo::IPinfo->new($access_token);
my $ip_address = '216.239.36.21';
my $details = $ipinfo->info($ip_address);
my $city = $details->city;  # Emeryville
my $loc = $details->loc;    # 37.8342,-122.2900

ASCII Art

+-------------------+       +-------------------+       +-------------------+
|                   |       |                   |       |                   |
|   Cron Job        | ----> |   Shell Script    | ----> |   Boto3 Script    |
|                   |       |                   |       |                   |
+-------------------+       +-------------------+       +-------------------+
        |                           |                           |
        |                           |                           |
        |                           |                           |
        v                           v                           v
+-------------------+       +-------------------+       +-------------------+
|                   |       |                   |       |                   |
|   Execute Shell   | ----> |   Execute Boto3   | ----> |   Interact with   |
|   Script          |       |   Script          |       |   SQLite Database |
|                   |       |                   |       |                   |
+-------------------+       +-------------------+       +-------------------+

Prompt

Other than lists, strings, tuples, dictionaries, integers, classes, functions, and metaclasses, what other data structures are in Python core without the use of module import statements?

Utility Functionality

lib 2to3

• https://github.com/python/cpython/issues/84540

Definition

def run_if_main(): # Your code here print("This code runs only if the script is executed directly.")

if name == "main": run_if_main()

Usage

another_module.py

from my_utils import run_if_main

Other code here

Call the utility function

run_if_main()