[pull] master from mozilla:master#1
Merged
pull[bot] merged 1583 commits intoappotry:masterfrom May 2, 2021
Merged
Conversation
Add CALLSTACK_INSTRUMENT environment variable
The test is passing locally, this is just a gentle code cleanup to force travis to rerun.
It's not in ff yet, but will be soon, and setting an unused config does not harm.
Check if the geckodriver process has children before attempting to access its Firefox child process. This will prevent getting an IndexError in case the geckodriver process has no child Firefox process because of an unexpected failure. Closes #626
Prevent IndexError in memory watchdog
Update to FF75, including new setting.
Co-authored-by: Steven Englehardt <englehardt@gmail.com>
Co-authored-by: Steven Englehardt <englehardt@gmail.com>
Fixing crawler
Currently, the HTTP response headers of the server are not logged when the server responds with an HTTP redirect. Wit this patch, they are saved as well.
* Make core changes reinstating Xvfb * Latest requirements Added pyvirtualdisplay but running pip-compile caused additional upgrades. * Default should not be headless * Fix flake8 * Revert "Latest requirements" This reverts commit 36989e9. * Manually add only pyvirtualdisplay * Parametrize test_simple_commands for two display modes. * flake8 * Rebalance tests test_[a-d, d-e] and test_c both taking 5 minutes each can be combined. Other tests hopefully taking ~10 min each. * Update crawler.py and demo.py * Add DISPLAY_MODE to sentry * flake8 * Add extra info about display_modes
Test whether at least the location header is properly saved by the http redirect instrumentation.
Added autopep8 pre-commit hook
Use the public suffix + 1 instead of the public suffix when comparing the domains in the crawl database with those in the profile history. Also, update an incorrectly formed query to the crawl database.
Make `PatchedGeckoDriverService` class subclass selenium.webdriver.firefox.service.Service instead of selenium.webdriver.common.service.Service, so that we only have to keep track of the changes in the `__init__()` method of the former class.
webdriver.switch_to.alert unlike most other variants of the switch_to API is not a function but a property. This led to TypeError:'Alert' object is not callable when there was actually an Alert to switch to. This PR fixes that behaviour.
This way the default config in index.js and the check in loggingdb.js match up
GitHub Action based on https://github.com/usha-mandya/SimpleWhaleDemo/blob/master/.github/workflows/github_registry.yml Everything on the master branch gets published as `:latest` and version tags (pattern `vx.y.z`) get published as `:x.y.z`
Restore stateful crawling support
Introduced `cleaned_js_instrument_settings` in BrowserParamsInternal to hold the expanded config dict. Propagating the `js_instrument_settings` through the extension as an object for as long as possible.
* Remove some unused files * Fix minor typos
* Renamed Browser to BrowserManagerHandler * Renamed TaskManager._issue_command to BrowserManagerHandle.execute_command_sequence * Fixing stuff * Apply suggestions from code review Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com> * tm to task_manager * Found and renamed only mention of in the docs Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Removing localtest.me As it has been highly unreliable when running local tests (returning DnsNotFound errors) * Fixing tests * Switched to localhost * Localtest.me to localhost
We can now generate documentation to a variety of display formats including HTML by using sphinx. With this new infrastructure we are now also able to generate documentation on readthedocs.io. Co-authored-by: jhabarsingh <jhabarsinghbhati23@gmail.com> Co-authored-by: Cyrus <cyruskarsan@gmail.com> Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com> Co-authored-by: Steven Englehardt <senglehardt@mozilla.com> Co-authored-by: ankushduacodes <61025943+ankushduacodes@users.noreply.github.com> Co-authored-by: Mollie Bakal <bakalm@umich.edu> Co-authored-by: MollieBakal <molliebakal@gmail.com> Co-authored-by: jhabarsingh <43932986+jhabarsingh@users.noreply.github.com> Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Created environment-rtd.yaml * Removed all non-sphinx dependencies
* Fix documentation module index Populate the module index by setting up Sphinx to automatically run sphinx-apidoc for every build. Also, move readthedocs dependencies under docs/ and make prune-environment.py automatically generate the environment-rtd.yaml file whenever we run repin.sh. * Fix black and mypy errors
Our default was to create a literal folder called ~ instead of writing to the homedir
* Re-enable test_profile_saved_when_launch_crashes Update `test_profile_saved_when_launch_crashes` so that it does not depend on the no longer supported proxy to make browser restarts fail. Instead, set the `FIREFOX_BINARY` environment variable to point to a wrong path. Also, fix a bug in `kill_browser_manager()`, which would cause OpenWPM to crash because of a `psutil.NoSuchProcess` exception and without archiving the browser profile, whenever a browser restart failed to launch geckodriver. Finally, make `kill_process_and_children()` use the timeout set via its arguments, which it previously ignored. * Update docstring of dump_profile Add a note for callers that they should make sure Firefox is closed, in order to prevent ending up with a corrupted database in the archived profile. * Update test_browser_profile_coverage Remove the buggy and outdated for loop that determined whether a url is expected to be missing from the places.sqlite database of the browser profile, as we have not observed any missing urls when running this test.
* Refactored BrowserManager into class * Completing refactor * Improved type annotations
* Combined log_directory and log_file to log_path * Updated documentation * Fixed tests
* Combined log_directory and log_file to log_path * Updated documentation * Fixed tests * Implemented test, need to change CSP * Extension logging restored and tested * Renamed extra to custom_params * Reverting stackdump changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )