Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local failure of "make test-lite" #53

Closed
martinholmer opened this issue Apr 23, 2024 · 4 comments
Closed

Local failure of "make test-lite" #53

martinholmer opened this issue Apr 23, 2024 · 4 comments
Assignees
@nikhilwoodruff
Labels
tests Test infrastructure and improvements

Comments

@martinholmer
Copy link
Collaborator

After updating the the repository and executing make install-lite, I tried to execute make test-lite on my computer. It doesn't seem right that the simplest test in an open-source repository should require a "token" to access data in a different (Policyengine-US) repository.

Here is what happens on my computer:

tax-microdata-benchmarking% make test-lite   
TEST_MODE=lite pytest . -v
============================= test session starts ==============================
platform darwin -- Python 3.9.13, pytest-7.4.0, pluggy-0.13.1 -- /Users/mrh/opt/anaconda3/bin/python
cachedir: .pytest_cache
rootdir: /Users/mrh/work/tax-microdata-benchmarking
plugins: anyio-3.5.0, dependency-0.5.1
collected 77 items                                                             

tests/test_basic_flat_file.py::test_flat_file_builds FAILED              [  1%]
tests/test_basic_flat_file.py::test_tc_variable_totals[FLPDYR] SKIPPED   [  2%]
tests/test_basic_flat_file.py::test_tc_variable_totals[MARS] SKIPPED     [  3%]
tests/test_basic_flat_file.py::test_tc_variable_totals[PT_SSTB_income] SKIPPED [  5%]
tests/test_basic_flat_file.py::test_tc_variable_totals[PT_ubia_property] SKIPPED [  6%]
tests/test_basic_flat_file.py::test_tc_variable_totals[XTOT] SKIPPED     [  7%]
tests/test_basic_flat_file.py::test_tc_variable_totals[age_head] SKIPPED [  9%]
tests/test_basic_flat_file.py::test_tc_variable_totals[age_spouse] SKIPPED [ 10%]
tests/test_basic_flat_file.py::test_tc_variable_totals[blind_head] SKIPPED [ 11%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00200] SKIPPED   [ 12%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00300] SKIPPED   [ 14%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00400] SKIPPED   [ 15%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00600] SKIPPED   [ 16%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00650] SKIPPED   [ 18%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00700] SKIPPED   [ 19%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00800] SKIPPED   [ 20%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e00900] SKIPPED   [ 22%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e01100] SKIPPED   [ 23%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e01400] SKIPPED   [ 24%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e01500] SKIPPED   [ 25%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e01700] SKIPPED   [ 27%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e02000] SKIPPED   [ 28%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e02300] SKIPPED   [ 29%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e02400] SKIPPED   [ 31%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03150] SKIPPED   [ 32%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03210] SKIPPED   [ 33%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03220] SKIPPED   [ 35%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03230] SKIPPED   [ 36%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03240] SKIPPED   [ 37%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03270] SKIPPED   [ 38%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03290] SKIPPED   [ 40%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03300] SKIPPED   [ 41%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03400] SKIPPED   [ 42%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e03500] SKIPPED   [ 44%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e07240] SKIPPED   [ 45%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e07260] SKIPPED   [ 46%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e07300] SKIPPED   [ 48%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e07400] SKIPPED   [ 49%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e07600] SKIPPED   [ 50%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e09700] SKIPPED   [ 51%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e09800] SKIPPED   [ 53%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e09900] SKIPPED   [ 54%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e11200] SKIPPED   [ 55%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e18400] SKIPPED   [ 57%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e19200] SKIPPED   [ 58%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e19800] SKIPPED   [ 59%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e20100] SKIPPED   [ 61%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e20400] SKIPPED   [ 62%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e24515] SKIPPED   [ 63%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e24518] SKIPPED   [ 64%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e26270] SKIPPED   [ 66%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e27200] SKIPPED   [ 67%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e32800] SKIPPED   [ 68%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e58990] SKIPPED   [ 70%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e62900] SKIPPED   [ 71%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e87521] SKIPPED   [ 72%]
tests/test_basic_flat_file.py::test_tc_variable_totals[e87530] SKIPPED   [ 74%]
tests/test_basic_flat_file.py::test_tc_variable_totals[f2441] SKIPPED    [ 75%]
tests/test_basic_flat_file.py::test_tc_variable_totals[f6251] SKIPPED    [ 76%]
tests/test_basic_flat_file.py::test_tc_variable_totals[g20500] SKIPPED   [ 77%]
tests/test_basic_flat_file.py::test_tc_variable_totals[housing_ben] SKIPPED [ 79%]
tests/test_basic_flat_file.py::test_tc_variable_totals[mcaid_ben] SKIPPED [ 80%]
tests/test_basic_flat_file.py::test_tc_variable_totals[mcare_ben] SKIPPED [ 81%]
tests/test_basic_flat_file.py::test_tc_variable_totals[n1820] SKIPPED    [ 83%]
tests/test_basic_flat_file.py::test_tc_variable_totals[n21] SKIPPED      [ 84%]
tests/test_basic_flat_file.py::test_tc_variable_totals[n24] SKIPPED      [ 85%]
tests/test_basic_flat_file.py::test_tc_variable_totals[nu06] SKIPPED     [ 87%]
tests/test_basic_flat_file.py::test_tc_variable_totals[nu13] SKIPPED     [ 88%]
tests/test_basic_flat_file.py::test_tc_variable_totals[nu18] SKIPPED     [ 89%]
tests/test_basic_flat_file.py::test_tc_variable_totals[other_ben] SKIPPED [ 90%]
tests/test_basic_flat_file.py::test_tc_variable_totals[p08000] SKIPPED   [ 92%]
tests/test_basic_flat_file.py::test_tc_variable_totals[p22250] SKIPPED   [ 93%]
tests/test_basic_flat_file.py::test_tc_variable_totals[snap_ben] SKIPPED [ 94%]
tests/test_basic_flat_file.py::test_tc_variable_totals[ssi_ben] SKIPPED  [ 96%]
tests/test_basic_flat_file.py::test_tc_variable_totals[tanf_ben] SKIPPED [ 97%]
tests/test_basic_flat_file.py::test_tc_variable_totals[vet_ben] SKIPPED  [ 98%]
tests/test_basic_flat_file.py::test_tc_variable_totals[wic_ben] SKIPPED  [100%]

=================================== FAILURES ===================================
____________________________ test_flat_file_builds _____________________________

    @pytest.mark.dependency()
    def test_flat_file_builds():
        from tax_microdata_benchmarking.create_flat_file import (
            create_stacked_flat_file,
        )
    
>       flat_file = create_stacked_flat_file(2021, reweight=test_mode == "full")

tests/test_basic_flat_file.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tax_microdata_benchmarking/create_flat_file.py:757: in create_stacked_flat_file
    puf_based_flat_file = create_flat_file(
tax_microdata_benchmarking/create_flat_file.py:659: in create_flat_file
    sim = Microsimulation(reform=taxcalc_extension, dataset=source_dataset)
../../opt/anaconda3/lib/python3.9/site-packages/policyengine_us/system.py:115: in __init__
    super().__init__(*args, **kwargs)
../../opt/anaconda3/lib/python3.9/site-packages/policyengine_core/simulations/simulation.py:156: in __init__
    self.dataset: Dataset = dataset(require=True)
../../opt/anaconda3/lib/python3.9/site-packages/policyengine_core/data/dataset.py:71: in __init__
    self.download()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <policyengine_us.data.datasets.puf.puf.PUF_2022 object at 0x7fbf3ef259d0>
url = 'https://api.github.com/repos/policyengine/non-public-microdata/releases/tags/puf-2022'

    def download(self, url: str = None):
        """Downloads a file to the dataset's file path.
    
        Args:
            url (str): The url to download.
        """
    
        if url is None:
            url = self.url
    
        if "POLICYENGINE_GITHUB_MICRODATA_AUTH_TOKEN" not in os.environ:
            auth_headers = {}
        else:
            auth_headers = {
                "Authorization": f"token {os.environ['POLICYENGINE_GITHUB_MICRODATA_AUTH_TOKEN']}",
            }
    
        # "release://" is a special protocol for downloading from GitHub releases
        # e.g. release://policyengine/policyengine-us/cps-2023/cps_2023.h5
        # release://org/repo/release_tag/file_path
        # Use the GitHub API to get the download URL for the release asset
    
        if url.startswith("release://"):
            org, repo, release_tag, file_path = url.split("/")[2:]
            url = f"https://api.github.com/repos/{org}/{repo}/releases/tags/{release_tag}"
            response = requests.get(url, headers=auth_headers)
            if response.status_code != 200:
>               raise ValueError(
                    f"Invalid response code {response.status_code} for url {url}."
                )
E               ValueError: Invalid response code 404 for url https://api.github.com/repos/policyengine/non-public-microdata/releases/tags/puf-2022.

../../opt/anaconda3/lib/python3.9/site-packages/policyengine_core/data/dataset.py:300: ValueError
----------------------------- Captured stdout call -----------------------------
Creating CPS flat file for 2021
Creating PUF flat file for 2021
=========================== short test summary info ============================
FAILED tests/test_basic_flat_file.py::test_flat_file_builds - ValueError: Invalid response code 404 for url https://api.github.com/repos/...
=================== 1 failed, 76 skipped in 94.41s (0:01:34) ===================
make: *** [test-lite] Error 1
@martinholmer martinholmer added the tests Test infrastructure and improvements label Apr 23, 2024
@nikhilwoodruff
Copy link
Collaborator

nikhilwoodruff commented Apr 23, 2024 •
edited

Martin- would appreciate your advice here but I really don't think there's any alternative if the test suite is going to actually test the PUF flat file data creation, given we cannot publish the PUF microdata. Even if we're not pulling it from PolicyEngine-US's repo, we will need to pull from a different repo because GitHub doesn't support private data releases on public repos.

In any case, I've just emailed you the token though.

@martinholmer
Copy link
Collaborator Author

@nikhilwoodruff, Thanks for sending me the token. Using it allowed me to run successfully the make test-lite command on my computer.

You're correct about the need for this given the privacy-status of the PUF data. However, it seems as if adding a note (say to the README.md file) that alerts people to the need for a token would be helpful. Otherwise, they will be puzzled about why the simplest test doesn't work.

Also, is it possible to define a narrower test that can be run without a token?
If so, the README.md file could point people to that option.

nikhilwoodruff added a commit that referenced this issue Apr 23, 2024
@nikhilwoodruff
Fix Local failure of "make test-lite" #53
a7a9cae
@nikhilwoodruff
Copy link
Collaborator

Thanks @martinholmer- I've added a note to the Readme in #52.

@martinholmer
Copy link
Collaborator Author

@nikhilwoodruff said:

I've added a note to the Readme in #52.

Thanks! That's perfect.

@martinholmer martinholmer mentioned this issue Apr 23, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikhilwoodruff nikhilwoodruff

Labels
tests Test infrastructure and improvements
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@martinholmer @nikhilwoodruff