-
-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support logging for multiple projects, log most warnings instead of raising them #327
Conversation
I am very happy for you to change the naming. This comes from a day when I had NO idea how logging worked |
I noticed I'm getting some hard errors (maybe segfaults) during tests sometimes. I initially thought it had something to do with my computer, but now I also see it popping up in CI (see linting / testing 3.7 ubuntu). Is that something you've seen before? I currently have no idea where it comes from |
Sure, will do |
* renamed starts_logging and combine with log.py * moved additional setup code to __init__.py * undo falsely blackened line
They were more frequent a while back, but I haven't gotten any in some time. If I remember correctly, the errors were related to errors in the Cython code that made it go outside array boundaries, which was hard to detect with the decorators that void that checking (which are in place for performance reasons). |
What version of Python are you using? It could be something related to dependencies (it fails for 3.7 and passes for 3.8, and the versions for SciPy and Pandas are different, for example). I wonder if this has something to do with the recent change in NumPy array sizes... |
I'm running on 3.8.12 locally, but I'm not getting the error every time, and also not on the same test. It does always seem to be failing when interacting with sqlite, sometimes using sqlite directly and sometimes through pandas. Not sure what the cause is though. |
@@ -107,12 +107,12 @@ def add_mode(self, mode: [str, Mode]): | |||
mode_id = self.__validate(mode) | |||
|
|||
if mode_id in self.modes: | |||
logger.warning("Mode already active for this link") | |||
self._logger.warning("Mode already active for this link") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that _logger is populated for the link is it? self.project.logger perhaps?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's a member of SafeClass
, so it is there :)
import time | ||
import re | ||
import requests | ||
from .osm_utils.osm_params import http_headers, memory | ||
from aequilibrae.parameters import Parameters | ||
from aequilibrae import logger | ||
from aequilibrae.context import get_logger |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
again - i don't think that get_logger is used here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's used on line 39: self.logger = logger or get_logger()
I'm also happy with this PR - a couple of questions/comments/suggestions thrown in. The number of CI tests is pretty big at the moment - might be worth rationalising them somewhat if they are getting in the way of actually getting things shipped instead of supporting that. |
Thanks @jamiecook for your review, I've processed your changes. I'm still looking at the SIGABRT errors, I have no idea where they come from. At my machine they popup sometimes, but only when running the full unit test suite. It seems that in CI it's pretty reproducible for python3.7 but I haven't been able to reproduce it in docker (perhaps stemming from me using python docker image instead of ubuntu). There also shouldn't be any significant difference between this branch and @jamiecook what do you mean when you say the number of CI tests is pretty big? the number of workflows / matrix size of the number of unittests? If it's the number of workflows/matrix size then this could be reduced, although given that they're mostly run in parallel I'm not sure if it would speed up things. If it's about the number of unittests then I disagree, it's the unittests that make it possible to refactor code such as I did with the multiple-projects PR. I do have some ideas to speed up the tests, but that would involve making more use of pytest. |
@elfjes - my comment was more on the number of workflows (currently 17) than the number of unittests. I agree on the utility of unittests. However any test-suite that can't be consistently relied upon can sometimes be more of a hinderance than a help for development. Having said that I'm not sure what to do here either - python 3.7 still has a year left before EOL, so should ideally be kept. sigh |
I've added a PR that should get rid of the first two failures from forks - #329 |
Nice. I do agree that it is important to not have regularly failing pipelines (and accept such a fact). So good stuff on that PR.
As a test I added a workflow that performs the tests in a container, which did not crash the tests. So while I still don't know what causes it exactly, it does seem to be some kind of binary incompatibility somewhere. My suggestion for now would be to run all (failing) workflows in a container when possible, which seems (n=1) to be mitigating the problem. We may want to have a separate pr that deals with the workflows more comprehensively. There are some other gains to be made (even if we keep the same number of workflows) for example, having the unit tests depend on the linting is not necessary I'd say, and I think the test setup can also be simplified. This could be done in #329 perhaps? |
#329 is already merged but yes a separate pr to deal with that would be better what do you mean “ performs tests in a container” I thought all gh actions were done using containers? Or are you saying using separate containers for individual states in a workflow? |
As it turns out (and I didn't know this either), actions are performed in a virtual machine, not a container. Which makes their specific environment more difficult to predict (and reproduce). By adding a container directive to the workflow, you can perform the action in a container (inside the vm). Have a look at the |
Oh yeah - that makes sense as to why each step is able to use filesystem artifacts from previous steps. I would have expected that VM would itself be a container though. Lets merge this and move discussion to your new branch - create now and mark it [WIP] if you'd like early feedback. |
* Updates shapely deprecated methods (#281) * Improves performance of triggers (#278) * documentation correction (#282) * INRETS and bpr2 VD functions (#273) * French INRETS VD function First implementation of the French INRETS Volume Delay function. Beta is not used but kept for better consistancy across VD functions. * BPR2 implementation First implementation of BPR2 Volume Delay function. This doubles beta above capacity so that fewer vehicles are affected when capacity is exceeded. Double Beta insteed of using a Beta' allow to use only 2 parameters as for other VD functions. Integration into QGIS GUI is also easier * Two small changes around the edges (#288) * Closes issue #289 and adds small feature * Fixes long-standing formatting issue * fixes traffic assignment tests (#293) * Follows the Wikimedia user-agent policy (#295) * Makes sure we follow the Wikimedia user agent policy https://meta.wikimedia.org/wiki/User-Agent_policy * Adresses issue #285 (#297) * Allows for skims to be set after the creation of traffic classes * Improvement on skim saving * Updates example with change in skim saving method * Fixes typo * Fixes creation of skim matrices on multi-class assignment * Fixes final skim for travel time and introduces basic testing * Fixes example with new skimming standards * cleans tests * Relocates and Fixes path saving tests * Updates tests * QGIS adjustments (#280) * Threading * Adds connection refresh capability * Stores binaries with all tests * Makes feather optional on import * Removes repeated code * Bumps up version * Adds doc version to index * Bumps version for new release * closes issue #303 * closes issue #303 (#304) * Fixes issue 307 * Improves database structure (#310) * Improves database structure * Improves database structure * numpy 1.22 is still not the default * CI workflow to generate binaries for different NumPy versions * Sets up better tests * Sets up better tests * numpy 1.22 is still not the default * Cleans test * Run black on all source files * Adding conda_forge_recipe * Support for multiple projects (#323) * Matrix from list (#325) * Pedro/new test fix (#322) * Speeding up OSM network import * only run actions that require secrets from the Aequilbrae account. (#329) * Support logging for multiple projects, log most warnings instead of raising them (#327) * one logger per project, log most warnings instead of raising them * Multiple changes * renamed starts_logging and combine with log.py * moved additional setup code to __init__.py * undo falsely blackened line * process feedback * cleanup ci setup for unittests - do errors still occur? * Small changes to the Delaunay line creation in trying to identify consistent breaking point Co-authored-by: Danilo Ebbinghaus <91709021+daniloebbinghaus@users.noreply.github.com> Co-authored-by: Arthur Evrard <83211842+Art-Ev@users.noreply.github.com> Co-authored-by: Jan Zill <jan.zill@veitchlister.com.au> Co-authored-by: Jan <jan.christoph.zill@gmail.com> Co-authored-by: Jamie Cook <jamie.cook@veitchlister.com.au> Co-authored-by: PelleK <elfjes@users.noreply.github.com> Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl> Co-authored-by: pveigadecamargo <pveigadecamargo@anl.gov>
* Updates shapely deprecated methods (#281) * Improves performance of triggers (#278) * documentation correction (#282) * INRETS and bpr2 VD functions (#273) * French INRETS VD function First implementation of the French INRETS Volume Delay function. Beta is not used but kept for better consistancy across VD functions. * BPR2 implementation First implementation of BPR2 Volume Delay function. This doubles beta above capacity so that fewer vehicles are affected when capacity is exceeded. Double Beta instead of using a Beta' allow to use only 2 parameters as for other VD functions. Integration into QGIS GUI is also easier * Fixes styles to expose test failures * Fixes styles to expose test failures * reverts white space changes in bpr * Fixes inrets tests * Two small changes around the edges (#288) * Adds the obvious field vot to the modes table and creates and adds a script for compiling that may be useful during development * Update tests/compile.py Co-authored-by: Jamie Cook <jamie.cook@veitchlister.com.au> * Fixes import * Closes issue #289 and adds small feature * Fixes long-standing formatting issue * fixes traffic assignment tests (#293) Brilliant work, @janzill That original test was really out of whack!!! * Follows the Wikimedia user-agent policy (#295) * Makes sure we follow the Wikimedia user agent policy https://meta.wikimedia.org/wiki/User-Agent_policy * Adresses issue #285 (#297) * Allows for skims to be set after the creation of traffic classes * Improvement on skim saving * Updates example with change in skim saving method * Fixes typo * Fixes creation of skim matrices on multi-class assignment * Fixes final skim for travel time and introduces basic testing * Fixes example with new skimming standards * cleans tests * Relocates and Fixes path saving tests * Updates tests * QGIS adjustments (#280) * Threading * Adds connection refresh capability * Stores binaries with all tests * Makes feather optional on import * Removes repeated code * Bumps up version * Adds doc version to index * Bumps version for new release * closes issue #303 * closes issue #303 (#304) * Fixes issue 307 * Improves database structure (#310) * Improves database structure * Improves database structure * numpy 1.22 is still not the default * CI workflow to generate binaries for different NumPy versions * Sets up better tests * Sets up better tests * numpy 1.22 is still not the default * use temp dir * guessing this is a case problem * guessing this is a case problem * guessing this is a case problem * it was a case issue * Cleans test * Run black on all source files * Update build_linux.yml * Adding conda_forge_recipe * Support for multiple projects (#323) * Matrix from list (#325) * Pedro/new test fix (#322) * Fixes linux building * Fixes linux building * Speeding up OSM network import * only run actions that require secrets from the Aequilbrae account. (#329) * Support logging for multiple projects, log most warnings instead of raising them (#327) * one logger per project, log most warnings instead of raising them * Multiple changes * renamed starts_logging and combine with log.py * moved additional setup code to __init__.py * undo falsely blackened line * process feedback * cleanup ci setup for unittests - do errors still occur? * Edit the correct ci file * we do need more requirements * run in docker to improve isolation * apt get update Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl> * Small changes to the Delaunay line creation in trying to identify consistent breaking point * Fix some warnings Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl> * Refactor tests to pytest (#331) * Refactor tests * Cleanup and some comments * fix tests * process feedback * Update tests/aequilibrae/conftest.py Co-authored-by: Jamie Cook <jimi.cook@gmail.com> * rescope session-scoped fixtures inside a class to class scope to avoid confusion * fix trailing whitespace Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl> Co-authored-by: Jamie Cook <jimi.cook@gmail.com> * change the check for secrets from repo_owner to has_secret (#333) * better deal with np.memmap open files (#335) Co-authored-by: Pelle <pellekoster@octagonal.nl> * Adding capability of importing and exporting networks in GMNS format (#330) * Pedro/fixes numpy changes (#342) * Fixes nan to num * Chipping along transit structure * Fixes tests * Uses new version of Numpy for Windows 3.10 only * address issues from review * unneeded Co-authored-by: pveigadecamargo <pveigadecamargo@anl.gov> Co-authored-by: Jamie Cook <jimi.cook@gmail.com> * Updates actions (#341) * Updates actions * Chipping along transit structure Co-authored-by: pveigadecamargo <pveigadecamargo@anl.gov> Co-authored-by: Jamie Cook <jamie.cook@veitchlister.com.au> * Fixes issue #344 (#345) * Fixes issue #344 Co-authored-by: François P <35044397+djfrancesco@users.noreply.github.com> * Bumps up version * Bumps up version (#346) Co-authored-by: pveigadecamargo <pveigadecamargo@anl.gov> * bug fix (#347) * Updates actions (#349) Co-authored-by: pveigadecamargo <pveigadecamargo@anl.gov> * Update Develop * stops using Kumi endpoint Co-authored-by: Danilo Ebbinghaus <91709021+daniloebbinghaus@users.noreply.github.com> Co-authored-by: Arthur Evrard <83211842+Art-Ev@users.noreply.github.com> Co-authored-by: Jan Zill <jan.zill@veitchlister.com.au> Co-authored-by: Jan <jan.christoph.zill@gmail.com> Co-authored-by: Jamie Cook <jamie.cook@veitchlister.com.au> Co-authored-by: PelleK <elfjes@users.noreply.github.com> Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl> Co-authored-by: pveigadecamargo <pveigadecamargo@anl.gov> Co-authored-by: Jamie Cook <jimi.cook@gmail.com> Co-authored-by: Pelle <pellekoster@octagonal.nl>
This PR tackles a couple of thing with regards to logging (fixes #324):
Project().logger
aequilibrae.context
module contains aget_logger
function that returns the currently active project's logger, or the global logger if there is no active projectwarnings.warn
calls have been replaced with the appropiatelogger.warning
call.global_logger
to avoid confusion. The logger is still available under the old namelogger
for backwards compatibilityconftest.py
that contains global fixturesOther fixes
noteworthy points/caveats:
aequilibrae.log
belong to the aequilibrae library/applicationparameters:52
. In order to log this message to the project logger, we need to know whether logging is enabled for this project, which we only know after the Parameters have been setup. I have circumvented it by logging the message to the global loggerOpen questions:
starts_logging
andStartsLogging
. I don't know the origin of this naming, but do you guys think it's still the right name?