[gha] Create release.yml

[vchrombie]

This commit adds the release github action which generates a package, tests it with different python versions, publishes the release on GitHub, and uploads the packages to PyPI whenever a tag is generated and pushed.

[coveralls]

Coverage remained the same at 96.994% when pulling 66c6470 on vchrombie:release into a34e088 on chaoss:master.

[vchrombie]

I think we should not merge this and wait till we fix the issues with the perceval script at least. We can test the workflow once everything is ready.

[vchrombie]

The PR was blocked because of two main reasons as mentioned in the #731 (comment).

* Install scripts

We have decided to change the script as an entry point as the script feature in poetry is not yet released and might take some more time. #741

* Install dev dependencies only

I wrote a small tool vchrombie/peodd that exports the dev-dependencies from pyproject.toml file to requirements.txt format. We can make use of this until poetry has the official feature of exporting only the dev-dependencies.

@sduenas, please review this when you are free and let me know your thoughts on the PR.

[sduenas]

The PR was blocked because of two main reasons as mentioned in the #731 (comment).

* Install scripts

We have decided to change the script as an entry point as the script feature in poetry is not yet released and might take some more time. #741

This is already merged.

* Install dev dependencies only

I wrote a small tool vchrombie/peodd that exports the dev-dependencies from pyproject.toml file to requirements.txt format. We can make use of this until poetry has the official feature of exporting only the dev-dependencies.

@sduenas, please review this when you are free and let me know your thoughts on the PR.

I think I didn't understand this part. I don't get why we need this when we create a package and we test it. The package should run with the dependencies that are already in the pyproject file. Can you explain this better?

[vchrombie]

Hello @sduenas

This is already merged.

Yes, this issue is now solved.

I think I didn't understand this part. I don't get why we need this when we create a package and we test it. The package should run with the dependencies that are already in the pyproject file. Can you explain this better?

When we build the package using poetry the dev-dependencies are ignored. I think this how it was designed to be. The dev-dependencies don't really matter to the users and it makes sense to me.

$ poetry build
Building perceval (0.17.4)
  - Building sdist
  - Built perceval-0.17.4.tar.gz
  - Building wheel
  - Built perceval-0.17.4-py3-none-any.whl

$ PACKAGE_NAME=`(cd dist && ls *whl | cut -f 1 -d "-")` && echo $PACKAGE_NAME
perceval

$ pip install --pre --find-links ./dist/ $PACKAGE_NAME
Looking in links: ./dist/
Collecting perceval
  Using cached perceval-0.17.7-py3-none-any.whl (211 kB)
Collecting urllib3>=1.22
...
Installing collected packages: urllib3, idna, certifi, chardet, requests, six, python-dateutil, grimoirelab-toolkit, PyJWT, pycparser, cffi, cryptography, soupsieve, beautifulsoup4, sgmllib3k, feedparser, dulwich, perceval
Successfully installed PyJWT-2.1.0 beautifulsoup4-4.9.3 certifi-2021.5.30 cffi-1.14.6 chardet-4.0.0 cryptography-3.4.7 dulwich-0.20.23 feedparser-6.0.8 grimoirelab-toolkit-0.2.0 idna-2.10 perceval-0.17.7 pycparser-2.20 python-dateutil-2.8.1 requests-2.25.1 sgmllib3k-1.0.0 six-1.16.0 soupsieve-2.2.1 urllib3-1.26.6

grimoirelab-perceval/dist/perceval-0.17.4/setup.cfg

install_requires = \
['PyJWT>=1.7.1,<2.0.0',
 'beautifulsoup4>=4.3.2,<5.0.0',
 'cryptography>=3.3.1,<4.0.0',
 'dulwich>=0.20.0,<0.21.0',
 'feedparser>=5.1.3,<6.0.0',
 'grimoirelab-toolkit>=0.1.12,<0.2.0',
 'python-dateutil>=2.6.0,<3.0.0',
 'requests>=2.7.0,<3.0.0',
 'urllib3>=1.22,<2.0']

So, now we install the package and there will be a lot of errors while testing as the dependencies which are required to run the tests are not installed.

======================================================================
ERROR: test_twitter (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_twitter
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/p0tt3r/dev/sources/grimoirelab-perceval/tests/test_twitter.py", line 33, in <module>
    import httpretty
ModuleNotFoundError: No module named 'httpretty'


----------------------------------------------------------------------
Ran 296 tests in 33.036s

FAILED (errors=31)

Poetry cannot explicitly install only the dev-dependencies. There was discussion around it and a couple of PRs are merged too but I think there is a lot of work still pending. Reference: python-poetry/poetry#2572

Another workaround could be exporting the dependencies. Again, Poetry cannot explicitly export only the dev-dependencies. There was a discussion around this too, but not sure if they would be interested to add this feature in the poetry export plugin. Reference: python-poetry/poetry#1441

So, to solve this issue, I wrote a small tool that exports the dev-dependencies to a requirements.txt format file. As the purpose is only for the CI, we can avoid the burden of maintaining the requirements in different files.

Please let me know your thoughts.

[sduenas]

I understand what you mean but for the tests when the package is built the dev-dependencies / test-dependencies should be the ones that are in the pyproject.yml. We will install those dependencies from that file in the repository.

For the CI tests - when someone submits a PR or something is merged in a branch - the dev-dependencies should the ones in poetry + the dependencies of the other grimoirelab components in git.

What I was saying is to have the latest in a separate requirements.txt file. For the CI, first, we will install all the dependencies in the pyproject.yml with poetry, and then, all of those in that requirements file with pip, probably. The requirements file can be named grimoirelab-dev-depencencies.txt.

For the build workflow, we will create the package, install it, and install the dev dependencies from the pyproject file.

Does it make sense to you?

[sduenas]

LGTM

[vchrombie]

Thanks @sduenas for merging the PR.

I wrote a small tool vchrombie/peodd that exports the dev-dependencies from pyproject.toml file to requirements.txt format. We can make use of this until poetry has the official feature of exporting only the dev-dependencies.

We would be using this untill we have some workaround with poetry.