fix: warn large resource issue#429

[Vinay26k]

PR for Issue #429

changes in the PR:

• implementation of _check_file_size(path) warns user about file > 1MB
• function call before calculating hashing of resource file

---

pipeline.yaml

tasks:
  - source: tasks.raw.get
    product: products/raw/get.csv
    params:
      resources_:
        file: conf.yaml
    .....

conf.yaml: dummy file with ~9MB of data
script used to test:

def get(resources_, product):
    print(" -------------- Debug --------------")
    print(open(resources_['file']).read()[:10])
    ....

output:
File too large. Resource <DIR-PATH>/conf.yaml[9.98MB] > 1MB

[edublancas]

looks good! can you add a test? please add it to the test_resources.py file

you can check for warnings using this: https://docs.pytest.org/en/6.2.x/warnings.html

and to simulate a big file, you can patch the os.stat call:

from ploomber.products import _resources

def test_process_resources_warns_on_large_file(monkeypatch):
    # this will cause the os.stat call in the _resources module to return 2E+6
    monkeypatch.setattr(_resources.os, 'stat', lambda _: 2E+6)
    # check warning is displayed

[edublancas]

thanks a lot for your contribution!