Skip to content

Commit

Permalink
Refactor to support Astro
Browse files Browse the repository at this point in the history
  • Loading branch information
jdddog committed Feb 25, 2024
1 parent c7e9331 commit 66ce99f
Show file tree
Hide file tree
Showing 252 changed files with 5,031 additions and 29,215 deletions.
10 changes: 2 additions & 8 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[run]
branch=True
source=observatory
source=observatory_platform

[report]
exclude_lines =
Expand All @@ -12,10 +12,4 @@ exclude_lines =
@abstract
ignore_errors = True
omit =
tests/*
observatory-api/observatory/api/client/api_client.py
observatory-api/observatory/api/client/configuration.py
observatory-api/observatory/api/client/exceptions.py
observatory-api/observatory/api/client/model_utils.py
observatory-api/observatory/api/client/rest.py
observatory-platform/observatory/platform/airflow/*
*/tests*
3 changes: 2 additions & 1 deletion .gitattributes
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
tests/fixtures/** filter=lfs diff=lfs merge=lfs -text
**/fixtures/** filter=lfs diff=lfs merge=lfs -text
docs/* linguist-documentation
notebooks/* linguist-documentation
*.csv filter=lfs diff=lfs merge=lfs -text
**/fixtures/**/* filter=lfs diff=lfs merge=lfs -text
25 changes: 6 additions & 19 deletions .github/workflows/publish-pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,36 +10,23 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.10

- name: Build packages
run: |
cd observatory-api
python3 setup.py sdist
cd ../observatory-platform
cp ../README.md .
python3 setup.py sdist
cd ../
- name: Publish observatory-api
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.PYPI_PASSWORD }}
packages_dir: observatory-api/dist/
python3 -m pip install --upgrade build
apt install python3.10-venv
python3 -m build --sdist
- name: Publish observatory-platform
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@master
with:
user: __token__
password: ${{ secrets.PYPI_PASSWORD }}
packages_dir: observatory-platform/dist/
packages_dir: ./dist/
10 changes: 4 additions & 6 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,26 +14,24 @@ jobs:

steps:
- name: Checkout ${{ matrix.python-version }}
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
lfs: true

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e observatory-api[tests] --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-no-providers-${{ matrix.python-version }}.txt
pip install -e observatory-platform[tests] --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-no-providers-${{ matrix.python-version }}.txt
pip install -e .[tests] --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-no-providers-${{ matrix.python-version }}.txt
- name: Check licenses
run: |
# stop the build if there are licensing issues
liccheck --sfile strategy.ini --rfile observatory-api/requirements.txt --level CAUTIOUS --reporting liccheck-output.txt --no-deps
liccheck --sfile strategy.ini --rfile observatory-platform/requirements.txt --level CAUTIOUS --reporting liccheck-output.txt --no-deps
liccheck
- name: Lint with flake8
run: |
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,6 @@ ChangeLog
/observatory-dags/observatory/dags/workflows/oapen_cloud_function.zip
docs/schemas
.env
/observatory-platform/build
/observatory-api/build
liccheck-output.txt
160 changes: 31 additions & 129 deletions LICENSES_THIRD_PARTY
Original file line number Diff line number Diff line change
@@ -1,140 +1,42 @@
Licenses of third party code included in this project are documented in this file as well as in individual source files.
# Code in observatory_platform/sandbox/sandbox_environment.py for running Airflow tasks comes from:
# * https://github.com/apache/airflow/blob/ffb472cf9e630bd70f51b74b0d0ea4ab98635572/airflow/cli/commands/task_command.py
# * https://github.com/apache/airflow/blob/master/docs/apache-airflow/best-practices.rst

The below files are included from https://github.com/keras-team/keras and have the following license:
# With the license:

- observatory_platform/utils/data_utils.py
- observatory_platform/utils/progbar_utils.py

# COPYRIGHT
#
# All contributions by François Chollet:
# Copyright (c) 2015 - 2019, François Chollet.
# All rights reserved.
#
# All contributions by Google:
# Copyright (c) 2015 - 2019, Google, Inc.
# All rights reserved.
#
# All contributions by Microsoft:
# Copyright (c) 2017 - 2019, Microsoft, Inc.
# All rights reserved.
#
# All other contributions:
# Copyright (c) 2015 - 2019, the respective contributors.
# All rights reserved.
#
# Each contributor holds copyright over their respective contributions.
# The project versioning (Git) records all such contribution source information.
#
# LICENSE
#
# The MIT License (MIT)
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

The below files are included from https://github.com/commoncrawl/cc-pyspark and have the following license:

- observatory_platform/common_crawl/cc_fetcher.py

# The MIT License (MIT)
#
# Copyright (c) 2017 Common Crawl
# Copyright (c) 2019 Curtin University
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# http://www.apache.org/licenses/LICENSE-2.0
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

The below file are included from https://github.com/fengwangPhysics/matplotlib-chord-diagram and have the following
license:

- observatory_platform/analysis/charts/chord.py
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# MIT License
#
# Copyright (c) 2017 Feng Wang
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

The below files are included from https://github.com/sphinx-doc/sphinx/tree/master/sphinx/templates/quickstart and have
the following license:
# Code in observatory_platform/sandbox/sftp_server is based on:
# * https://github.com/rspivak/sftpserver/blob/master/src/sftpserver/__init__.py

- observatory-platform/observatory/platform/cli/templates/generate_project/sphinx-quickstart/conf.py_t
- observatory-platform/observatory/platform/cli/templates/generate_project/sphinx-quickstart/make.bat.new_t
- observatory-platform/observatory/platform/cli/templates/generate_project/sphinx-quickstart/Makefile.new_t
- observatory-platform/observatory/platform/cli/templates/generate_project/sphinx-quickstart/root_doc.rst_t
# With the license:

# License for Sphinx
# ==================
#
# Copyright (c) 2007-2021 by the Sphinx team (see AUTHORS file).
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
# Copyright 2021-2024 Curtin University
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# http://www.apache.org/licenses/LICENSE-2.0
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
23 changes: 16 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
![Observatory Platform](https://raw.githubusercontent.com/The-Academic-Observatory/observatory-platform/develop/logo.jpg)

The Observatory Platform is an environment for fetching, processing and analysing data to understand how well
universities operate as Open Knowledge Institutions.
The Observatory Platform is a set of dependencies used by the Curtin Open Knowledge Initiative (COKI) for running its
Airflow based workflows to fetch, process and analyse bibliometric datasets.

The Observatory Platform is built with Apache Airflow. The workflows for the project can be seen in the
[Academic Observatory Workflows](https://github.com/The-Academic-Observatory/academic-observatory-workflows)
and [OAeBU Workflows](https://github.com/The-Academic-Observatory/oaebu-workflows) projects.
The workflows for the project can be seen in at:
* [Academic Observatory Workflows](https://github.com/The-Academic-Observatory/academic-observatory-workflows)
* [OAeBU Workflows](https://github.com/The-Academic-Observatory/oaebu-workflows)

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Python Version](https://img.shields.io/badge/python-3.10-blue)](https://img.shields.io/badge/python-3.10-blue)
Expand All @@ -15,8 +15,17 @@ and [OAeBU Workflows](https://github.com/The-Academic-Observatory/oaebu-workflow
[![codecov](https://codecov.io/gh/The-Academic-Observatory/observatory-platform/branch/develop/graph/badge.svg)](https://codecov.io/gh/The-Academic-Observatory/observatory-platform)
[![DOI](https://zenodo.org/badge/227744539.svg)](https://zenodo.org/badge/latestdoi/227744539)

## Documentation
For more detailed documentation about the Observatory Platform see the Read the Docs website [https://observatory-platform.readthedocs.io](https://observatory-platform.readthedocs.io)
## Dependencies
Observatory Platform supports Python 3.10, Ubuntu Linux 22.04 and MacOS 10.14, on x86 architecture.

System dependencies:
* Python 3.10
* Pip
* virtualenv
* Google Cloud SDK (optional): https://cloud.google.com/sdk/docs/install-sdk

## Python Package Reference
See the Read the Docs website for documentation on the Python package [https://observatory-platform.readthedocs.io](https://observatory-platform.readthedocs.io)

## Dependent Repositories
The Observatory Platform is a dependency for other repositories developed and maintained by [The Academic Observatory](https://github.com/The-Academic-Observatory):
Expand Down
34 changes: 0 additions & 34 deletions bin/generate-api-client.sh

This file was deleted.

Loading

0 comments on commit 66ce99f

Please sign in to comment.