Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

coverage combine throws sqlite3.OperationalError: unable to open database file #992

Closed
shadycuz opened this issue May 25, 2020 · 5 comments
Closed

coverage combine throws sqlite3.OperationalError: unable to open database file #992

shadycuz opened this issue May 25, 2020 · 5 comments
Labels
bug Something isn't working

Comments

@shadycuz
Copy link

Describe the bug
When trying to combine two coverage data files I get sqlite exception.

To Reproduce

Try to combine two data files inside a github action?

  1. What version of Python are you using?
    CPython (3.8.3)

  2. What version of coverage.py are you using?

-- sys -------------------------------------------------------
                        version: 5.1
                       coverage: /opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                   configs_read: /home/runner/work/crawlers/crawlers/.coveragerc
                    config_file: /home/runner/work/crawlers/crawlers/.coveragerc
                config_contents: '[run]\nbranch=True\n\nomit =\n\t*/venv/*\n\t*/test/*\n'
                      data_file: -none-
                         python: 3.8.3 (default, May 14 2020, 09:47:01) [GCC 7.5.0]
                       platform: Linux-5.3.0-1020-azure-x86_64-with-glibc2.2.5
                 implementation: CPython
                     executable: /opt/hostedtoolcache/Python/3.8.3/x64/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 2618
                            cwd: /home/runner/work/crawlers/crawlers
                           path: /opt/hostedtoolcache/Python/3.8.3/x64/bin
                                 /opt/hostedtoolcache/Python/3.8.3/x64/lib/python38.zip
                                 /opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8
                                 /opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/lib-dynload
                                 /opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages
                    environment: -none-
                   command_line: /opt/hostedtoolcache/Python/3.8.3/x64/bin/coverage debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.22.0
             sqlite3_temp_store: 0
        sqlite3_compile_options: COMPILER=gcc-7.4.0
                                 ENABLE_COLUMN_METADATA
                                 ENABLE_DBSTAT_VTAB
                                 ENABLE_FTS3
                                 ENABLE_FTS3_PARENTHESIS
                                 ENABLE_FTS3_TOKENIZER
                                 ENABLE_FTS4
                                 ENABLE_FTS5
                                 ENABLE_JSON1
                                 ENABLE_LOAD_EXTENSION
                                 ENABLE_PREUPDATE_HOOK
                                 ENABLE_RTREE
                                 ENABLE_SESSION
                                 ENABLE_STMTVTAB
                                 ENABLE_UNLOCK_NOTIFY
                                 ENABLE_UPDATE_DELETE_LIMIT
                                 HAVE_ISNAN
                                 LIKE_DOESNT_MATCH_BLOBS
                                 MAX_SCHEMA_RETRY=25
                                 MAX_VARIABLE_NUMBER=250000
                                 OMIT_LOOKASIDE
                                 SECURE_DELETE
                                 SOUNDEX
                                 TEMP_STORE=1
                                 THREADSAFE=1
  1. What versions of what packages do you have installed? The output of pip freeze is helpful.
astroid==2.4.1
backports.functools-lru-cache==1.6.1
beautifulsoup4==4.8.2
bs4==0.0.1
certifi==2020.4.5.1
chardet==3.0.4
click==7.1.1
coverage==5.1
Flask==1.1.2
idna==2.9
isort==4.3.21
itsdangerous==1.1.0
Jinja2==2.11.2
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
mccabe==0.6.1
numpy==1.18.3
pandas==1.0.3
pylint==2.5.2
pymongo==3.10.1
python-dateutil==2.8.1
pytz==2020.1
requests==2.23.0
rope==0.17.0
selenium==3.141.0
six==1.14.0
soupsieve==1.9.5
toml==0.10.0
urllib3==1.25.8
Werkzeug==1.0.1
wrapt==1.12.1
  1. What code are you running? Give us a specific commit of a specific repo that we can check out.
    Private repo, but some pretty generic tests
  2. What commands did you run?
    coverage combine
    Expected behavior
    I expected the data files to be combined and the command to return a 0 exit code.

Additional context

Running this on github actions but coverage run and coverage xml work fine.

Maybe the sql database isn't created until another command like coverage run or coverage report is run?

Files in the directory...

Run ls -la
total 160
drwxr-xr-x 9 runner docker  4096 May 25 15:20 .
drwxr-xr-x 3 runner docker  4096 May 25 15:19 ..
drwxr-xr-x 2 runner docker  4096 May 25 15:20 .coverage.functional
drwxr-xr-x 2 runner docker  4096 May 25 15:20 .coverage.unit
-rw-r--r-- 1 runner docker    68 May 25 15:19 .coveragerc
drwxr-xr-x 8 runner docker  4096 May 25 15:19 .git
drwxr-xr-x 3 runner docker  4096 May 25 15:19 .github
-rw-r--r-- 1 runner docker    89 May 25 15:19 .gitignore
-rw-r--r-- 1 runner docker 18424 May 25 15:19 .pylintrc
-rw-r--r-- 1 runner docker 76015 May 25 15:19 Output.zip
-rw-r--r-- 1 runner docker 11800 May 25 15:19 README.md
-rw-r--r-- 1 runner docker    61 May 25 15:19 WebsiteList.txt
-rw-r--r-- 1 runner docker   506 May 25 15:19 requirements.txt
drwxr-xr-x 2 runner docker  4096 May 25 15:19 scripts
drwxr-xr-x 4 runner docker  4096 May 25 15:19 test
@shadycuz shadycuz added the bug Something isn't working label May 25, 2020
@shadycuz
Copy link
Author

forgot the output of the error...

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.3/x64/bin/coverage", line 8, in <module>
    sys.exit(main())
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/cmdline.py", line 827, in main
    status = CoverageScript().command_line(argv)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/cmdline.py", line 561, in command_line
    self.coverage.combine(data_dirs, strict=True)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/control.py", line 687, in combine
    combine_parallel_data(self._data, aliases=aliases, data_paths=data_paths, strict=strict)
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/data.py", line 110, in combine_parallel_data
    new_data.read()
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/sqldata.py", line 742, in read
    with self._connect():       # TODO: doesn't look right
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/sqldata.py", line 297, in _connect
    self._open_db()
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/sqldata.py", line 265, in _open_db
    self._read_db()
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/sqldata.py", line 269, in _read_db
    with self._dbs[get_thread_id()] as db:
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/sqldata.py", line 1026, in __enter__
    self._connect()
  File "/opt/hostedtoolcache/Python/3.8.3/x64/lib/python3.8/site-packages/coverage/sqldata.py", line 1008, in _connect
    self.con = sqlite3.connect(filename, check_same_thread=False)
sqlite3.OperationalError: unable to open database file
##[error]Process completed with exit code 1.

@epicserve
Copy link

I'm having this same exact issue, but in Bitbucket Pipelines.

@epicserve
Copy link

@shadycuz, did you happen to figure out a workaround?

@shadycuz
Copy link
Author

shadycuz commented Jun 20, 2020 •
edited
Loading

@epicserve nope =/, because we are also using codecov, we can just upload both tests seperately and it will combine them for us. So fixing this wasn't high priority.

@epicserve
Copy link

@shadycuz, I was able to get this to work on Bitbucket Pipelines but I had do a couple of things. Mainly I had to stop using my custom built docker image that was basically using docker within docker and switched to using a docker image off of docker hub. This is the general gist of my configuration for Bitbucket Pipelines.

@nedbat, I think you can probably close this issue because this seems to be a bug related more with how CI systems work.

Makefile:

.PHONY: check_py
check_py: ## Runs tests with coverage and uses codecov to give a coverage report
	$(PYTHON_CMD_PREFIX) coverage run -p --concurrency=multiprocessing manage.py test --parallel -v 3 --no-js

.PHONY: check_js
check_js: ## Runs selenium JS tests
	$(PYTHON_CMD_PREFIX) coverage run -p --concurrency=multiprocessing ./manage.py test -v 3 --no-py

.PHONY: push_to_codecov
push_to_codecov: ## Pushes code coverage to Codecov. Only used for Bitbucket Pipelines
	echo "Before combining ..." && ls -la .coverage_data/
	coverage combine .coverage_data/
	echo "After combining ..." && ls -la .coverage_data/
	coverage xml
	echo "Directory contents ..." && ls -la && echo `pwd`
	codecov --token=<token>

bitbucket-pipelines.yml

pipelines:
  default:
    - parallel:
      - step:
          <<: *LINT
      - step:
          <<: *BUILD
    - parallel:
        - step:
          - curl -s -o /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py && python /tmp/get-pip.py
          - pip install -U awscli
          - make pull-image
          - |
              mkdir -p .coverage_data/ && docker run --rm \
                -v $BITBUCKET_CLONE_DIR/.coverage_data/:/code/.coverage_data \
                $DOCKER_REGISTRY/pears:$BITBUCKET_COMMIT make check_py
        - step:
          - curl -s -o /tmp/get-pip.py https://bootstrap.pypa.io/get-pip.py && python /tmp/get-pip.py
          - pip install -U awscli
          - docker run --name selenium -d -p 4444:4444 selenium/standalone-chrome
          - make pull-image
          - |
              mkdir -p .coverage_data/ && docker run --rm \
                --link selenium:selenium/standalone-chrome \
                -v $BITBUCKET_CLONE_DIR/.coverage_data/:/code/.coverage_data \
                $DOCKER_REGISTRY/pears:$BITBUCKET_COMMIT make check_js

    - step:
      name: Report Code Coverage
      caches:
        - pip
      artifacts:
        - .coverage_data/**
      image: python:3-buster
      script:
        - pip install coverage~=5.1 codecov~=2.1.1
        - make push_to_codecov
    - step:
        <<: *DEPLOY

@platipo platipo mentioned this issue Oct 7, 2020
Merged
@nedbat nedbat closed this as completed May 3, 2021
@felipetio felipetio mentioned this issue May 24, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nedbat @epicserve @shadycuz