Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix "update cassettes" step #4591

Merged
merged 4 commits into from
Jun 6, 2023
Merged

Fix "update cassettes" step #4591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c7b95da
Fix updated cassettes step
waynehamadi Jun 5, 2023
0ba1083
Clarifications
Pwuts Jun 6, 2023
1d7a186
Use github.ref_name instead of github.ref
Pwuts Jun 6, 2023
8803f5d
Fix duplicate runs on `master`
Pwuts Jun 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
109 changes: 41 additions & 68 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ concurrency:
jobs:
lint:
# eliminate duplicate runs on master
if: github.event_name == 'push' || github.ref_name != 'master' || (github.event.pull_request.head.repo.fork == (github.event_name == 'pull_request_target'))
if: github.event_name == 'push' || github.base_ref != 'master' || (github.event.pull_request.head.repo.fork == (github.event_name == 'pull_request_target'))

runs-on: ubuntu-latest
env:
Expand All @@ -37,14 +37,15 @@ jobs:
with:
python-version: ${{ env.min-python-version }}

- name: Set Date
run: echo "DATE=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
- id: get_date
name: Get date
run: echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT

- name: Cache Python packages
- name: Set up Python dependency cache
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}-${{ env.DATE }}
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}-${{ steps.get_date.outputs.date }}

- name: Install dependencies
run: |
Expand Down Expand Up @@ -73,7 +74,7 @@ jobs:

test:
# eliminate duplicate runs on master
if: github.event_name == 'push' || github.ref_name != 'master' || (github.event.pull_request.head.repo.fork == (github.event_name == 'pull_request_target'))
if: github.event_name == 'push' || github.base_ref != 'master' || (github.event.pull_request.head.repo.fork == (github.event_name == 'pull_request_target'))

permissions:
# Gives the action the necessary permissions for publishing new
Expand All @@ -90,16 +91,20 @@ jobs:
python-version: ["3.10"]

steps:
- name: Check out repository
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ github.event.pull_request.head.ref }}
repository: ${{ github.event.pull_request.head.repo.full_name }}
submodules: true

- id: checkout_cassettes
name: Check out cassettes
- name: Configure git user Auto-GPT-Bot
run: |
git config --global user.name "Auto-GPT-Bot"
git config --global user.email "github-bot@agpt.co"
Pwuts marked this conversation as resolved.
Show resolved Hide resolved

- name: Checkout cassettes
if: ${{ startsWith(github.event_name, 'pull_request') }}
run: |
cassette_branch="${{ github.event.pull_request.user.login }}-${{ github.event.pull_request.head.ref }}"
Expand All @@ -111,43 +116,37 @@ jobs:

git checkout $cassette_branch

if git merge --no-commit --no-ff ${{ github.event.pull_request.base.ref }}; then
echo "Using cassettes from mirror branch, synced to upstream branch '${{ github.event.pull_request.base.ref }}'"
else
echo "Could not merge upstream changes to cassettes. Using cassettes from ${{ github.event.pull_request.base.ref }}."
git merge --abort
git checkout ${{ github.event.pull_request.base.ref }}

# Delete branch to prevent conflict when re-creating it
git branch -D $cassette_branch
fi
echo "cassette_branch=$(git branch --show-current)" >> $GITHUB_OUTPUT
# Pick non-conflicting cassette updates from the base branch
git merge --no-commit --strategy-option=ours origin/${{ github.event.pull_request.base.ref }}
echo "Using cassettes from mirror branch '$cassette_branch'," \
"synced to upstream branch '${{ github.event.pull_request.base.ref }}'."
else
echo "Branch '$cassette_branch' does not exist in cassette submodule."\
"Using cassettes from ${{ github.event.pull_request.base.ref }}."
echo "cassette_branch=${{ github.event.pull_request.base.ref }}" >> $GITHUB_OUTPUT
git checkout -b $cassette_branch
Pwuts marked this conversation as resolved.
Show resolved Hide resolved
echo "Branch '$cassette_branch' does not exist in cassette submodule." \
"Using cassettes from '${{ github.event.pull_request.base.ref }}'."
fi

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Set Date
run: echo "DATE=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
- id: get_date
name: Get date
run: echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT

- name: Cache Python packages
- name: Set up Python dependency cache
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}-${{ env.DATE }}
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}-${{ steps.get_date.outputs.date }}

- name: Install dependencies
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt

- name: Run pytest tests with coverage
- name: Run pytest with coverage
run: |
pytest -n auto --cov=autogpt --cov-report term-missing --cov-branch --cov-report xml --cov-report term
python tests/integration/challenges/utils/build_current_score.py
Expand All @@ -162,10 +161,9 @@ jobs:

- id: setup_git_auth
name: Set up git token authentication
# Cassettes may be pushed even when tests fail
if: success() || failure()
run: |
git config --global user.name "Auto-GPT-Bot"
git config --global user.email "github-bot@agpt.co"

config_key="http.${{ github.server_url }}/.extraheader"
base64_pat=$(echo -n "pat:${{ secrets.PAT_REVIEW }}" | base64 -w0)

Expand All @@ -186,58 +184,39 @@ jobs:
if ! git diff --quiet $score_file; then
git add $score_file
git commit -m "Update challenge scores"
git push origin HEAD:${{ github.ref }}
git push origin HEAD:${{ github.ref_name }}
else
echo "The challenge scores didn't change."
fi

- id: push_cassettes
name: Push updated cassettes
# For pull requests, push updated cassettes even when tests fail
if: github.event_name == 'push' || success() || failure()
run: |
if [ "${{ startsWith(github.event_name, 'pull_request') }}" = "true" ]; then
is_pull_request=true
cassette_branch="${{ github.event.pull_request.user.login }}-${{ github.event.pull_request.head.ref }}"
cassette_source_branch="${{ steps.checkout_cassettes.outputs.cassette_branch }}"
base_branch="${{ github.event.pull_request.base.ref }}"
else
current_branch=$(echo ${{ github.ref }} | sed -e "s/refs\/heads\///g")
cassette_branch=$current_branch
cassette_branch="${{ github.ref_name }}"
fi

cd tests/Auto-GPT-test-cassettes
Pwuts marked this conversation as resolved.
Show resolved Hide resolved
git fetch origin $cassette_source_branch:$cassette_source_branch

# Commit & push changes to cassettes if any
if ! git diff --quiet $cassette_source_branch --; then
if [ "$cassette_branch" != "$cassette_source_branch" ]; then
git checkout -b $cassette_branch
fi
if ! git diff --quiet; then
git add .
git commit -m "Auto-update cassettes"

if [ $is_pull_request ]; then
git push --force origin HEAD:$cassette_branch
else
git push origin HEAD:$cassette_branch
fi

cd ../..
if [ $is_pull_request ]; then
git fetch origin $base_branch
cassette_diff=$(git diff origin/$base_branch)
else
git push origin HEAD:$cassette_branch
if [ ! $is_pull_request ]; then
cd ../..
git add tests/Auto-GPT-test-cassettes
git commit -m "Update cassette submodule"
git push origin HEAD:$current_branch
git push origin HEAD:$cassette_branch
fi
else
echo "No cassette changes to commit"
fi

if [ -n "$cassette_diff" ]; then
echo "updated=true" >> $GITHUB_OUTPUT
else
echo "updated=false" >> $GITHUB_OUTPUT
echo "No cassette changes to commit"
fi

- name: Post Set up git token auth
Expand All @@ -246,7 +225,7 @@ jobs:
git config --unset-all '${{ steps.setup_git_auth.outputs.config_key }}'
git submodule foreach git config --unset-all '${{ steps.setup_git_auth.outputs.config_key }}'

- name: Apply or remove behaviour change label and comment on PR
- name: Apply "behaviour change" label and comment on PR
if: ${{ startsWith(github.event_name, 'pull_request') }}
run: |
PR_NUMBER=${{ github.event.pull_request.number }}
Expand All @@ -263,10 +242,4 @@ jobs:

echo $TOKEN | gh auth login --with-token
gh api repos/$REPO/issues/$PR_NUMBER/comments -X POST -F body="You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged."
else
echo "Removing label..."
curl -X DELETE \
-H "Authorization: Bearer $TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/repos/$REPO/issues/$PR_NUMBER/labels/behaviour%20change
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why ? because I realized this doesn't work properly, and it's very hard to make it work properly. Let me explain:

  • you run your tests
  • you get new cassettes and it says that you changed the behaviour
  • you're surprised because it wasn't intended.
  • you change your code
  • you push again. Unfortunately at that time we still have your old cassettes, so we can't really tell if you went back to normal. If we pick the cassettes of master, then the tests are way slower.

I will think about it more. It's already pretty good to be able to detect behaviour change, unfortunately, we can't detect if the behaviour returns to normal compared to master.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can detect which parts of the cassettes are actually being used, we can reset cassettes when their last entry is not used during a test run. (assuming new requests are always added to the end of a cassette)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's a good trick

fi
3 changes: 2 additions & 1 deletion tests/integration/challenges/basic_abilities/test_write_file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import pytest
from pytest_mock import MockerFixture

from autogpt.agent import Agent
from autogpt.commands.file_operations import read_file
Expand All @@ -17,7 +18,7 @@
@challenge
def test_write_file(
writer_agent: Agent,
patched_api_requestor: None,
patched_api_requestor: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
config: Config,
level_to_run: int,
Expand Down
3 changes: 2 additions & 1 deletion tests/integration/challenges/information_retrieval/test_information_retrieval_challenge_a.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import pytest
from pytest_mock import MockerFixture

from autogpt.commands.file_operations import read_file
from autogpt.config import Config
Expand All @@ -19,7 +20,7 @@
def test_information_retrieval_challenge_a(
information_retrieval_agents: Agent,
monkeypatch: pytest.MonkeyPatch,
patched_api_requestor: None,
patched_api_requestor: MockerFixture,
config: Config,
level_to_run: int,
) -> None:
Expand Down
3 changes: 2 additions & 1 deletion tests/integration/challenges/information_retrieval/test_information_retrieval_challenge_b.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import contextlib

import pytest
from pytest_mock import MockerFixture

from autogpt.agent import Agent
from autogpt.commands.file_operations import read_file
Expand All @@ -20,7 +21,7 @@
def test_information_retrieval_challenge_b(
get_nobel_prize_agent: Agent,
monkeypatch: pytest.MonkeyPatch,
patched_api_requestor: None,
patched_api_requestor: MockerFixture,
level_to_run: int,
config: Config,
) -> None:
Expand Down
2 changes: 2 additions & 0 deletions tests/integration/challenges/kubernetes/test_kubernetes_template_challenge_a.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import pytest
import yaml
from pytest_mock import MockerFixture

from autogpt.agent import Agent
from autogpt.commands.file_operations import read_file
Expand All @@ -19,6 +20,7 @@
def test_kubernetes_template_challenge_a(
kubernetes_agent: Agent,
monkeypatch: pytest.MonkeyPatch,
patched_api_requestor: MockerFixture,
config: Config,
level_to_run: int,
) -> None:
Expand Down
3 changes: 2 additions & 1 deletion tests/integration/challenges/memory/test_memory_challenge_a.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import pytest
from pytest_mock import MockerFixture

from autogpt.agent import Agent
from autogpt.commands.file_operations import read_file, write_to_file
Expand All @@ -15,7 +16,7 @@
@challenge
def test_memory_challenge_a(
memory_management_agent: Agent,
patched_api_requestor: None,
patched_api_requestor: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
config: Config,
level_to_run: int,
Expand Down
3 changes: 2 additions & 1 deletion tests/integration/challenges/memory/test_memory_challenge_b.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import pytest
from pytest_mock import MockerFixture

from autogpt.agent import Agent
from autogpt.commands.file_operations import read_file, write_to_file
Expand All @@ -17,7 +18,7 @@
@challenge
def test_memory_challenge_b(
memory_management_agent: Agent,
patched_api_requestor: None,
patched_api_requestor: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
config: Config,
level_to_run: int,
Expand Down
3 changes: 2 additions & 1 deletion tests/integration/challenges/memory/test_memory_challenge_c.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import pytest
from pytest_mock import MockerFixture

from autogpt.agent import Agent
from autogpt.commands.file_operations import read_file, write_to_file
Expand All @@ -18,7 +19,7 @@
@challenge
def test_memory_challenge_c(
memory_management_agent: Agent,
patched_api_requestor: None,
patched_api_requestor: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
config: Config,
level_to_run: int,
Expand Down