jobs

jobs

Provides routines to request and sync data on database.

---

JobStatus Objects

class JobStatus(Enum)

Enum to control job status.

---

PENDING

Job is queued and waiting to be processed.

---

FAILED

Job request failed.

---

INVALID

Job request returned item had incorrect fields.

---

CORRUPTED

Job with incorrect start date.

---

COMPLETED

Job successfully completed.

---

get_label

def get_label() -> str

Returns:

a string label for each status.

---

SerieStationData Objects

@dataclass
class SerieStationData()

Data class to receive Series Job data when starting a job.

---

station_code

Station code which data will be requested.

---

start_date

Start date which data will be requested

---

end_date

End date which data will be requested

---

get_token

def get_token() -> Token.Token

Authenticate and return access token.

Returns:

Valid token for requesition

---

check_resource

def check_resource(resource: HidroResource) -> None

Checks each HidroResource and request/update them.

Arguments:

• resource: Current Resource to check, insert and update.

Returns:

Nothing.

---

check_stations_jobs

def check_stations_jobs() -> None

Checks if there is Stations entries on database. Request all national stations by UF if there isnt.

---

check_series_job

def check_series_job(job_config: JobConfig) -> None

Checks an JobConfig. Create jobs if there is no entries and start pending jobs requisition.

---

create_series_jobs

def create_series_jobs(stations_data: List[SerieStationData],
                       job_config: JobConfig) -> None

Creates Series Jobs for each SerieStationData received for a given JobConfig. Preprocess all years from "Start Date" to "End Date" that will become a job request.

---

trigger_job

def trigger_job(job_config: JobConfig) -> None

Triggers an Thread Worker for each pending or falied job entrie in DB for a given JobConfig.

---

handle_job

def handle_job(job: HidroJob, job_config: JobConfig) -> None

Request data of an HidroJob. Validate data on success return, and convert to ORM model before writing on Queue.

---

db_writer

def db_writer() -> None

Single Writer Thread running during an Job. Consumes an Queue writen by each worker and write data in batches.

---

write_data

def write_data(job_config: JobConfig, jobs: List[HidroJob],
               hidro_data: dict) -> float

Insert data into DB and update the jobs as well.

---

validate_data

def validate_data(job_config: JobConfig, items: dict,
                  job: HidroJob) -> (HidroJob, dict)

Validate returned data by the API.

---

data_to_model_orm

def data_to_model_orm(job_config: JobConfig, hidro_data: dict)

Convert returned data by the API into the correspondent ORM Model of the job.