Scrubber sample code now handles "singular" tokens correctly

[0dB]

Scrubber sample code now handles tokens correctly that also appear by their own and not just in spans with other tokens.

When adding a token to the scrubber list like "two" that also exists as an individual term not just together with other tokens like "two sentences" the original code failed because the while loop consumed the whole span consisting of just one token. Fixed.

[ceteri]

@0dB ,

Could you make the following change in the .github/workflows/ci.yml file in your branch, then re-submit this PR?

A few lines need to be commented out:

name: CI

on: [push, pull_request, workflow_dispatch]

jobs:
#  pre-commit:
#    name: Run pre-commit
#    runs-on: ubuntu-latest
#    steps:
#      - uses: actions/checkout@v2
#      - uses: actions/setup-python@v2
#      - uses: pre-commit/action@v2.0.0

  test:
    name: Tests for Python ${{ matrix.python-version }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ['3.7', '3.8', '3.9', '3.10']
      fail-fast: false
#    needs: pre-commit

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          pip install -e .
          spacy download en_core_web_sm
          pip install pre-commit pytest

      - name: Run tests
        run: |
          pytest

GitHub currently is failing on the CI / Run pre-commit step, and we need to disable this step before a PR can be tested and merge.

And I want to be sure to give you credit for the PR in our Git history of the project.

[0dB]

Done, but I also see you are fighting with some issues; I'm afraid I am no help with those.

[ceteri]

Many thanks!
And no worries about the CI issues, we've got lots of devops experience here -- when GitHub is running correctly :)

FWIW, the CI actions built by GitHub for pre-commit hooks when through 3 versions; however, last year they archived that project and starting referring people to a different external project. GitHub doing something a bit strange with Py 3.11 when one attempts to use those archived actions. We just need to cut over to the newer ones.

Although we were blocked because once the CI actions on GitHub reach an error, there's really no way to back out without deleting or updating a PR.