Skip to content

Commit

Permalink
fix: solve the test issue (#20)
Browse files Browse the repository at this point in the history
  • Loading branch information
12rambau committed Feb 15, 2024
2 parents d251437 + f8e3c96 commit a9fc6b7
Show file tree
Hide file tree
Showing 5 changed files with 122 additions and 9 deletions.
3 changes: 2 additions & 1 deletion .github/workflows/unit.yaml
Expand Up @@ -8,7 +8,8 @@ on:
pull_request:

env:
EARTHENGINE_TOKEN: ${{ secrets.EARTHENGINE_TOKEN }}
EARTHENGINE_SERVICE_ACCOUNT: ${{ secrets.EARTHENGINE_SERVICE_ACCOUNT }}
EARTHENGINE_PROJECT: ${{ secrets.EARTHENGINE_PROJECT }}

jobs:
lint:
Expand Down
71 changes: 66 additions & 5 deletions docs/usage.rst
Expand Up @@ -27,11 +27,6 @@ To do so, the lib will provide a number of connection methods that will hopefull
All the methods presented in this section will fallback to a regular ``ee.Initialize()`` if the environment parameter are not found.
This means that you can use this plugin in your local environment without having to change anything as long as the ``ee`` module is installed and that you already run once ``ee.Authenticate()``.

Private Token
^^^^^^^^^^^^^

The first method is to use a private token. This is the easiest way to connect to GEE in a CI/CD environment.

.. danger::

Never forget that this method can potentially expose your personal credential to GEE so take some saety precaution before starting:
Expand All @@ -41,6 +36,11 @@ The first method is to use a private token. This is the easiest way to connect t
- make sure the account you are using will have access to all the assets you need to run your tests
- create small tests that will run quickly to make sure you don't overload your own GEE account with concurrent tasks

Private Token
^^^^^^^^^^^^^

The first method is to use a private token. This is the easiest way to connect to GEE in a CI/CD environment.

First authenticate to GEE API in your local computer using ``ee.Authenticate()``.

Then copy the ``credentials`` file content. This file is located in a different folder depending on the platform you use:
Expand All @@ -60,6 +60,8 @@ Here is a github action example:
#. Then to :guilabel:`secretes and variables` -> :guilabel:`Actions`
#. In this page, set a :guilabel:`new repository secret` with the name ``EARTHENGINE_TOKEN`` and paste the content of your ``credentials`` file in the value field.

Since earthengine-api v0.1.370, it's not possible to use EE without providing a GCS project bucket. Save this value in a `EARTHENGINE_PROJECT` variable, it will be used in the method.

To make the variable available in your CI environment, you will need to add the following line in your action `.yaml` file:

.. code-block:: yaml
Expand All @@ -68,9 +70,16 @@ To make the variable available in your CI environment, you will need to add the
env:
EARTHENGINE_TOKEN: ${{ secrets.EARTHENGINE_TOKEN }}
EARTHENGINE_PROJECT: ${{ secrets.EARTHENGINE_PROJECT }}
# The rest of your tests configuration
When working in your local environment export a ``EARTHENGINE_PROJECT`` variable as well:

.. code-block:: console
export EARTHENGINE_PROJECT=ee-community
Finally you need to configure the ``pytest`` execution environment itself. Add the following line in your ``conftest.py`` file:

.. code-block:: python
Expand All @@ -85,6 +94,58 @@ Finally you need to configure the ``pytest`` execution environment itself. Add t
You are now ready to make API calls within your tests!

Service account
^^^^^^^^^^^^^^^

.. warning::

This documentation assumes that you already have a Google cloud service account and that you have generated an API key for it. If not, please refer to Google own `documentation <https://cloud.google.com/iam/docs/keys-create-delete>` to proceed.

Paste this content of the `private-key.json` in your CI/CD environment in a ``EARTHENGINE_SERVICE_ACCOUNT`` variable.

Here is a github action example:

.. thumbnail:: _static/github_env_var.png
:title: Github action environment variable setup

#. First go to the :guilabel:`settings`` of your Github repository
#. Then to :guilabel:`secretes and variables` -> :guilabel:`Actions`
#. In this page, set a :guilabel:`new repository secret` with the name ``EARTHENGINE_SERVICE_ACCOUNT`` and paste the content of your ``credentials`` file in the value field.

Currently when the earthengine-api is Initialized using a service account, the name of the associated cloud project is not detectable. It will prevent the initialization of the test folder generated from `pytest-gee`. To avoid this issue the method rely also on a ``EARTHENGINE_PROJECT`` env variable where you can set the name of your project.

To make the variable available in your CI environment, you will need to add the following line in your action `.yaml` file:

.. code-block:: yaml
# .github/action.yaml
env:
EARTHENGINE_SERVICE_ACCOUNT: ${{ secrets.EARTHENGINE_SERVICE_ACCOUNT }}
EARTHENGINE_PROJECT: ${{ secrets.EARTHENGINE_PROJECT }}
# The rest of your tests configuration
When working in your local environment export a ``EARTHENGINE_PROJECT`` variable as well:

.. code-block:: console
export EARTHENGINE_PROJECT=ee-community
Finally you need to configure the ``pytest`` execution environment itself. Add the following line in your ``conftest.py`` file:

.. code-block:: python
# conftest.py
import pytest_gee
def pytest_configure():
pytest_gee.init_ee_from_service_account()
You are now ready to make API calls within your tests!

Generate a test file tree in GEE
--------------------------------

Expand Down
43 changes: 42 additions & 1 deletion pytest_gee/__init__.py
@@ -1,7 +1,9 @@
"""The init file of the package."""
from __future__ import annotations

import json
import os
import tempfile
from pathlib import Path
from typing import Union

Expand Down Expand Up @@ -37,9 +39,48 @@ def init_ee_from_token():
credential_file_path = credential_folder_path / "credentials"
credential_file_path.write_text(ee_token)

project_id = os.environ.get("EARTHENGINE_PROJECT", ee.data._cloud_api_user_project)
if project_id is None:
raise ValueError(
"The project name cannot be detected."
"Please set the EARTHENGINE_PROJECT environment variable."
)

# if the user is in local development the authentication should
# already be available
ee.Initialize(http_transport=httplib2.Http())
ee.Initialize(project=project_id, http_transport=httplib2.Http())


def init_ee_from_service_account():
"""Initialize earth engine according using a service account.
The environment used to run the tests need to have a EARTHENGINE_SERVICE_ACCOUNT variable.
The content of this variable must be the copy of a personal credential file that you can generate from the google cloud console.
Note:
As all init method of ``pytest-gee``, this method will fallback to a regular ``ee.Initialize`` using the ``EARTHENGINE_PROJECT`` environment variable.
"""
if "EARTHENGINE_SERVICE_ACCOUNT" in os.environ:
# extract the environment variables data
private_key = os.environ["EARTHENGINE_SERVICE_ACCOUNT"]
ee_user = json.loads(private_key)["client_email"]

# connect to GEE using a temp file to avoid writing the key to disk
with tempfile.TemporaryDirectory() as temp_dir:
file = Path(temp_dir) / "private_key.json"
file.write_text(private_key)
credentials = ee.ServiceAccountCredentials(ee_user, str(file))
ee.Initialize(credentials=credentials, http_transport=httplib2.Http())

elif "EARTHENGINE_PROJECT" in os.environ:
# if the user is in local development the authentication should already be available
# we simply need to use the provided project name
ee.Initialize(project=os.environ["EARTHENGINE_PROJECT"], http_transport=httplib2.Http())

else:
raise ValueError(
"EARTHENGINE_SERVICE_ACCOUNT or EARTHENGINE_PROJECT environment variable is missing"
)


def wait(task: Union[ee.batch.Task, str], timeout: int = 5 * 60) -> str:
Expand Down
11 changes: 10 additions & 1 deletion pytest_gee/plugin.py
@@ -1,6 +1,7 @@
"""A pytest plugin to build a GEE environment for a test session."""
from __future__ import annotations

import os
import uuid
from pathlib import Path

Expand All @@ -19,7 +20,15 @@ def gee_hash():
@pytest.fixture(scope="session")
def gee_folder_root():
"""Link to the root folder of the connected account."""
return Path(ee.data.getAssetRoots()[0]["id"])
# The credential information cannot be reached from
# the ee API as reported in https://issuetracker.google.com/issues/325020447
project_id = os.environ.get("EARTHENGINE_PROJECT", ee.data._cloud_api_user_project)
if project_id is None:
raise ValueError(
"The project name cannot be detected."
"Please set the EARTHENGINE_PROJECT environment variable."
)
return Path(f"projects/{project_id}/assets")


@pytest.fixture(scope="session")
Expand Down
3 changes: 2 additions & 1 deletion tests/conftest.py
@@ -1,4 +1,5 @@
"""Pytest session configuration."""

import ee
import pytest

Expand All @@ -7,7 +8,7 @@

def pytest_configure():
"""Init GEE in the test environment."""
pytest_gee.init_ee_from_token()
pytest_gee.init_ee_from_service_account()


@pytest.fixture(scope="session")
Expand Down

0 comments on commit a9fc6b7

Please sign in to comment.