Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Simplify logging via custom context manager text writer
- python standard library's logging package doesn't work for this use case
- use case:
- log output from multiple ListCreator instances running on different threads to DIFFERENT files
- tried running 2 instances of ListCreator on 2 different threads
- program scraping logic executes properly
- HOWEVER, content of log files is scrambled
- instead of thread 1 writing only to file 1 and thread 2 writing only to file 2:
- thread 1 writes to both file 1 AND file 2
- thread 2 writes to both file 2 AND file 1
- the pattern of a specific thread writing to different files was not readily obvious
- it DID seem like the first thread wrote only to its specified file UNTIL the second thread was created, after which the threads mixed their output
- logging package claims to be thread safe and available for use on multiple threads without additional configuration
- however, no variation of attribute changes, object creation, or logging.basicConfig() modification seemed to avoid the problem of writing to multiple files instead of the ONE designated file for that specific thread
- problem might be because logging package enables logging to all modules in a python package by making logging GLOBAL (?)
- logging may be global on a single python process (?)
- this would explain why both threads wrote to both open files instead of the file specified for their own thread
- ESSENTIALLY
- the logging from thread 1 initially sets the output to be file 1
- logging from thread 1 writes to only file 1
- thread 2 is created and the logging package sets thread 2 to write to file 2
- HOWEVER, since logging is global, thread 2 is essentially ADDING file 2 to the "list of logging output locations," therefore
- thread 2 writes to file 2 AND file 1
- thread 1 writes to file 1 AND file 2
- BECAUSE: logging package NOW takes anything that should be logged and logs it to EVERY LOCATION that is added to the "list of logging output locations" using the .addHandler() logging method
- the thread safe claim by logging package makes sense for a multi-threaded application that needs to log ALL info into the SAME file(s) EVEN with application running on multiple threads
- however, this use case is different and the goal is to NOT write the info to the same file(s) across different threads, but to write to only a SPECIFIC file specified by its specific thread
- workaround: provide the output location to any function that logged any information
- used @contextlib.contextmanager (to avoid manually creating __enter__() and __exit__() methods) for using a FUNCTION instead of a file like resource with the "with" keyword
- trying to run
```
with open(filename, 'a', encoding='utf-8') if log_file else sys.stdout as logging_output_location:
logging_output_location.writelines(content)
```
- did not seem like best practice, since trying to run these lines more than once resulted in:
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file.
```
- this might be okay in typical use cases if user exits the python interpreter after running ListCreator().create_list_for() method ONCE
- but might cause problems if user tries to run method again OR tries to print() anything in the same python process (does not restart interpreter)
- also tried:
```
with open(sys.stdout) as f:
f.write('some text')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected str, bytes or os.PathLike object, not _io.TextIOWrapper
```
- tried to "yield" a file object from a function without the @contextlib.contextmanager decorator:
```
def stdout_writer():
yield sys.stdout
def file_writer(filename):
with open(filename, 'a', encoding='utf-8') as file:
yield file
with file_writer(filename) if some_truthy_value else stdout_writer() as out:
out.write('haha')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __enter__
```
- decorating both functions with @contextlib.contextmanager addressed AttributeError: __enter__
```
@contextlib.contextmanager
def stdout_writer():
yield sys.stdout
@contextlib.contextmanager
def file_writer(filename):
with open(filename, 'a', encoding='utf-8') as file:
yield file
with file_writer(filename) if True else stdout_writer() as out:
out.write('haha')
with file_writer(filename) if False else stdout_writer() as out:
out.write('haha')
haha4
```
- as seen above, running the .write(content) method to the provided writer object always returns the NUMBER of bytes written
- the .writelines(content) method does NOT display number of bytes written to user
- HOWEVER, the .writelines(content) method does NOT add a newline to the end of every write, so '\n' needed to be manually added to all logging statements
- resources:
- logging package
- https://realpython.com/python-logging/#other-configuration-methods
- https://stackoverflow.com/questions/11232230/logging-to-two-files-with-different-settings/11233293
- https://stackoverflow.com/questions/15199816/python-logging-multiple-files-using-the-same-logger
- https://stackoverflow.com/questions/24816456/python-logging-wont-shutdown
- https://docs.python.org/3/library/logging.html
- https://docs.python.org/3/howto/logging-cookbook.html#:~:text=Although%20logging%20is%20thread-safe,across%20multiple%20processes%20in%20Python.
- https://docs.python.org/3/howto/logging-cookbook.html#multiple-handlers-and-formatters
- https://docs.python.org/3/library/logging.html#logging.getLogger
- sys.stdout, print(), and formatting strings
- https://stackoverflow.com/questions/3263672/the-difference-between-sys-stdout-write-and-print
- https://stackoverflow.com/questions/493386/how-to-print-without-newline-or-space?rq=1
- https://stackoverflow.com/questions/12377473/python-write-versus-writelines-and-concatenated-strings
- time package
- https://docs.python.org/3/library/time.html#time.strftime
- https://en.wikipedia.org/wiki/ISO_8601
- contextlib package
- https://dbader.org/blog/python-context-managers-and-with-statement
- https://docs.python.org/3/library/contextlib.html
- https://pymotw.com/2/contextlib/
- https://www.geeksforgeeks.org/context-manager-using-contextmanager-decorator/- Loading branch information
1 parent
7134c3e
commit 82a0129