Skip to content

Python

Jump to bottom
BK Jackson edited this page Jan 4, 2024 · 205 revisions

Python Subpages

Effective Python for Data Science

Efficient Python Tricks and Tools for Data Scientists [Class mixins over inheritance for enhanced modularity] (https://khuyentran1401.github.io/Efficient_Python_tricks_and_tools_for_data_scientists/Chapter1/class.html#use-mixins-over-inheritance-for-enhanced-modularity) - basically a trick for adding data or model IO as a class that is inherited by your main model class

Python Automation

pyautomate - CLI tool to automate any set of tasks. You describe a state machine in a config file and use the CLI to go to a certain state. pyautomate will automatically find out how to get there using the config file you provided.

Python Fire - Make command line interfaces (CLIs) from any Python object

Python Fire - github
Fire Getting Started Guide
Example of calling Fire on a function:

import fire

def hello(name="World"):
  return "Hello %s!" % name

if __name__ == '__main__':
  fire.Fire(hello)

Then, from the command line, you can run:

python hello.py  # Hello World!
python hello.py --name=David  # Hello David!
python hello.py --help  # Shows usage information.

Setting and getting environment variables

Use the dotenv package, which allows you to easily store these variables in a file that is not in source control (as long as you keep the line .env in your .gitignore file!). You can then reference these variables as environment variables in your application with
os.environ.get('VARIABLE_NAME').
View .env with
print(open(os.path.join(PROJ_ROOT, '.env')).read())

Convenience function to get the absolute path to a file in a directory from that same file

def script_path(filename):
    """
    A convenience function to get the absolute path to a file in this
    tutorial's directory. This allows the tutorial to be launched from any
    directory.

    """
    import os

    filepath = os.path.join(os.path.dirname(__file__))
    return os.path.join(filepath, filename)

Accessing Documentation for Python Functions

help(pickle.dump)

The ? only works in ipython, not python3:

pickle.dump?

To view the source code use ??. Does not work with builtin functions or methods, generally because the object is implemented in a language other than Python, such as C.

somefunction??

Access a web API and print the json output to the console

# print_user_agent.py
import requests

json = requests.get('http://httpbin.org/user-agent').json()
print(json['user-agent'])

Lambda functions

id = lambda nmodes, t: int((nmodes * t[0]) + t[1])  

$id(nmodes, t)

Create a sorted list of tuples with sorted() and a lambda function

The returned data value is 2nd in a list. That is data[1] = pair[1].

genrelist = [(genre, median)\
              for genre, median\
              in sorted(data, key=lambda pair: pair[1])])

Broadcasting

a = np.random.rand(10000, 1000)
b = np.arange(10000)

c = a + b

timeit -n 10 c = a + np.tile(b.reshape((-1, 1)), (1, 1000))

timeit -n 10 c = a + b.reshape((-1, 1))

array_equal(a + np.tile(b.reshape((-1, 1)), (1, 1000)), a + b.reshape((-1, 1)))

Fancy indexing with np.take

b = np.take(a, ind, axis=0)

Compress for boolean indexing

b = np.compress(ind, a, axis=0)

Formatting print strings and passing variables

print("Spaces %d. tabs %d. lines %d" % (spaces, tabs, lines))  
  
msg = "{0} loves {1}.".format(name, language)  
  
d = datetime.date(2004, 9, 8)  
f"{d} was a {d:%A}, we started the mailing list back then."  
'2004-09-08 was a Wednesday, we started the mailing list back then.'  

answer = 42  
print(f"The answer is {answer}")

Using finally for exception cleanup

try:  
    fobj = open("hello.txt", "w")  
    res = 12 / 0  
except ZeroDivisionError:  
    print("We have an error in division")  
finally:  
    fobj.close()  
    print("Closing the file object.")  

We have an error in division
Closing the file object.

While Loops

Use while loops when you don't know how many times you will iterate.

def countdown(n):
  """Print a countdown from n to 0."""   
  i = n  
  while i >= 0:  
    print i 
    i -= 1

countdown(5)

Searching for filenames with glob

csv_list = glob.glob('/some/path/**/*.csv', recursive=True)

Reading multiple files in one line

Using Pandas

import glob
import pandas as pd
df = pd.concat([pd.read_csv(f, encoding='latin1') for f in glob.glob('data*.csv'), ignore_index=True])

Using dask

import dask.dataframe as dd
df = dd.read_csv('*.csv')

Using os

import os
files = os.listdir()
# ['data_blah.csv', 'more_data.csv', 'yet_another.csv', 'lots_of_data.csv']

Decorators

Python Decorators: Syntactic Artificial Sweetener - From Chris Wellons, Null Program, 3/8/2019

Lambda, map, and filter

Lambda, Map, and Filter in Python

Learning Programming with Python

A primer on *args, **kwargs, decorators for Data Scientists
The Google Python Style Guide
Porting Python 2 code to Python 3 Includes handy, easy-to-skim tables with equivalent Python 2 and 3 commands.
Automate the Boring Stuff with Python
Problem Solving with Algorithms and Data Structures in Python
The Hitchhiker’s Guide to Python Includes coding style, OOP, web dev, more.
Python progression path - From apprentice to guru
Separate IO from Algorithms Tips for creating easy-to-maintain code.
Python Generator comprehensions
Awesome Python A curated list of awesome Python frameworks, libraries, software and resources.
Python Decorators
16 Python Tricks To Learn Before You Write Your Next Code

Click

Writing Python Command-Line Tools With Click - Part 1
Mastering Click: Writing Advanced Python Command-Line Apps - Part 2

Speeding up Python

Why Python is Slow: Looking Under the Hood
Speeding up Python with Numba - Examples
Numpy performance tricks

Testing

Improve Your Python: Understanding Unit Testing By Jeff Knupp.
pytest Python testing tool that helps you write better programs.

Python and Twitter

Tweepy

Sphinx for Python (& other code) documentation

Sphinx Home
First Steps with Sphinx Tutorial Includes installation info.
Math Support for Sphinx
Selecting an HTML Theme A theme is a collection of HTML templates, stylesheet(s) and other static files. See this page for built-in theme examples also.
Jinja/Sphinx Templating Sphinx uses the Jinja templating engine for its HTML templates. Jinja is a text-based engine, and inspired by Django templates.
Sampledoc Tutorial For getting started with Sphinx.
Pygments Python syntax highlighter.
Generate PDFs with LaTeXBuilder This builder produces a bunch of LaTeX files in the output directory.
reStructuredText Docs For the reStructuredText plaintext markup syntax and parser system.
Documentation using Sphinx and ReadTheDocs.org
Sphinx for Python Documentation - a detailed tutorial

Python on Windows

Reading and Writing Files

Note to Self: Saving Images with PIL (Python Imaging Library)

scipy.misc.toimage followed by PIL Image.save() is nearly 2x faster than skimage.io.imsave or scipy.misc.imsave. Also, skimage.io.imsave and scipy.misc.imsave do not easily pass parameters to underlying image save methods, including pil plugin. (Added: 3/16/2016)

Saving PIL image to TIFF format:

  • Following TIFF compression options throw error "encoder error -2 when writing image file": jpeg, tif_jpeg, tiff_ccitt, tiff_raw_16, group3, group4, tiff_adobe_deflate, tiff_thunderscan, tiff_deflate, tiff_sgilog, tiff_sgilog24
  • Only TIFF compression options that work for input 150 KB image: Default/None (14,792 KB), raw (14,792 KB), tiff_lzw (184 KB, no detail info), packbits (477 KB, no detail info)

Clone this wiki locally