Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Circleci project setup #291

Merged
merged 34 commits into from
Nov 21, 2023
Merged

Circleci project setup #291

merged 34 commits into from
Nov 21, 2023

Conversation

apetenchea
Copy link
Member

@apetenchea apetenchea commented Oct 6, 2023

This PR configures the CircleCI build: https://app.circleci.com/pipelines/github/ArangoDB-Community/python-arango

See description of Anthony's PR which implements part of the functionality.

Python Versions

We aim to support the Python versions that come with the last two Ubuntu LTS releases.

  • Ubuntu 22.04.2 - Python 3.10.6
  • Ubuntu 20.04.6 - Python 3.8.2

As new Ubuntu LTS versions are being released, we'll update the config accordingly.
Additionally, we run a couple of tests with the latest available Python version. These are not mandatory to pass, but should be there to alert us in case something gets deprecated or we're using something that's too version specific.

ArangoDB versions

We offer support for all the officially supported ArangoDB versions.

Remaining Github Workflows

The Github Workflows that were used to test various Python versions and run the single server tests were removed.
The following actions remain as part of the current Github Workflow:

  • docs - building, testing and HTML generation
  • CodeQL
  • PyPi (for releases)

Although we could move them to CircleCI, these are very lightweight and can might as well run on Github for now.

Additional Changes

  • Some of the tests had to be adapted in order to become usable in all the current testing combinations.
  • The starter.sh script is now available for easier local testing.
  • Due to some differences in 3.10 and 3.11, I had to create different config files for starting ArangoDB. The current single.conf and cluster.conf correspond to 3.11+ versions.
  • The FallbackHostResolver has been introduced as the default host resolver for the ArangoClient. I will explain the motivation below, but this was needed in order to run the tests reliably in a cluster. Previously, we've had the RoundRobinResolver, which would change the host quite often, in order to create some kind of load balancing. However, when multiple coordinators are involved, this can become a problem. First, it introduces subtle races. If a coordinator creates a database, the whole process has to go through the agency, during which the other coordinators are not aware of the newly created database. The Round Robin approach may cause consecutive requests to fail with 'database not found'. Secondly, this approach can sometimes make things slower by causing even more network traffic. This is counter-intuitive, but when dealing with streaming transactions, the other coordinators will identify the original coordinator which started the transaction and forward all transaction-related requests, essentially rendering the apparent load balancing useless. Lastly, it is currently the case with our driver that requests are synchronous. This means the next request is never sent unless the current one gets a response back. If a user wants to parallelize work, the usual approach would be to use multiple Python processes involving multiple clients. In that case, they would take care of load balancing themselves (by shuffling the list of coordinators between processes, for example).

To be improved

Code Coverage (see DE-701). Previously, we had limited coverage reporting, based on single-server tests. Although a very good first step, I feel like this just gave us the illusion of having code coverage. Ideally, we need to unify all the coverage reports from our CircleCI jobs into one.

@apetenchea apetenchea self-assigned this Oct 6, 2023
@apetenchea apetenchea marked this pull request as ready for review October 17, 2023 10:42
@aMahanna aMahanna mentioned this pull request Nov 9, 2023
@apetenchea apetenchea added the Ready to Merge Pull request is ready to merge label Nov 17, 2023
@aMahanna aMahanna self-requested a review November 21, 2023 02:40
@aMahanna aMahanna merged commit 52c7469 into main Nov 21, 2023
52 checks passed
@aMahanna aMahanna deleted the circleci-project-setup branch November 21, 2023 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ready to Merge Pull request is ready to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants