-
Notifications
You must be signed in to change notification settings - Fork 3
CU-869aa22g2 Add ElasticSearch bits from working_with_cogstack #123
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mart-r
wants to merge
40
commits into
main
Choose a base branch
from
CU-869aa22g2-add-es-from-wwc
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
40 commits
Select commit
Hold shift + click to select a range
8af57d8
CU-869aa22g2: Add ES requirement
mart-r 64f95ad
CU-869aa22g2: Add initial README
mart-r fcf4d96
CU-869aa22g2: Add (slightly) converted credentials
mart-r bfdcf82
CU-869aa22g2: Add cogstack module ported from WWC
mart-r 2332a63
CU-869aa22g2: Add search template notebook
mart-r 4dc92aa
CU-869aa22g2: Add ipython dependency (for cogstack and notebook)
mart-r 4f2d463
CU-869aa22g2: Add tqdm dependency
mart-r 61b20f0
CU-869aa22g2: Add pandas dependency
mart-r ec2a780
CU-869aa22g2: Add a few initial tests
mart-r 0322164
CU-869aa22g2: Remove a bunch of extra whitespace
mart-r 696e669
CU-869aa22g2: Add ruff dependecny
mart-r 90d339b
CU-869aa22g2: Run ruff on cogstack module
mart-r 77b3f3d
CU-869aa22g2: Move to primitives collections for type hinting
mart-r fe1656f
CU-869aa22g2: Some further linting changes
mart-r 780a3fd
CU-869aa22g2: Refactor cogstack module to make a little more sense
mart-r 403a2e8
CU-869aa22g2: Rename some methods for better descriptions
mart-r 2fd8507
CU-869aa22g2: Rename a method name for better descriptions
mart-r 7e04913
CU-869aa22g2: Minor whitespace fix
mart-r 62b60ca
CU-869aa22g2: Remove path add in notebook
mart-r 92aaabd
CU-869aa22g2: Remove unused import from notebook
mart-r 651a55b
CU-869aa22g2: Import username and password from credentials in case t…
mart-r e98f647
CU-869aa22g2: Add nbconvert dev-dependency
mart-r 4bd6246
CU-869aa22g2: Add default indices to get fields for
mart-r ab5dee8
CU-869aa22g2: Improve error handling (avoid hiding stack trace)
mart-r ab8da2c
CU-869aa22g2: Add default indices in notebook example
mart-r 4b77182
CU-869aa22g2: Update progress bar handling during exception handling
mart-r 3a604cf
CU-869aa22g2: Add default indices in notebook examples
mart-r 921d682
CU-869aa22g2: Add data folder
mart-r bd4f5bd
CU-869aa22g2 Fix data folder in notebook
mart-r b1790e0
CU-869aa22g2: Add initial notebook tests
mart-r 42eb024
CU-869aa22g2: Simplify test slightly
mart-r e570521
CU-869aa22g2: Remove test-time debug output
mart-r 35cce29
CU-869aa22g2: Add assertion and removal of data file created by notebook
mart-r a0c6fa3
CU-869aa22g2: Add initial workflow
mart-r 95fffd1
CU-869aa22g2: Fix workflow working directory
mart-r fdea8d8
CU-869aa22g2: Add OpenSearch dependency
mart-r 329f047
CU-869aa22g2: Allow OpenSearch to be used instead of ES
mart-r 379eb06
CU-869aa22g2: Add missing ES/OS helpers import
mart-r 349b7f3
CU-869aa22g2: Fix typo in variable name
mart-r 96857f0
CU-869aa22g2: Fix test time mocking
mart-r File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
name: cogstack-es - Test | ||
|
||
on: | ||
push: | ||
branches: [ main ] | ||
pull_request: | ||
paths: | ||
- 'cogstack-es/**' | ||
- '.github/workflows/cogstack-es**' | ||
|
||
defaults: | ||
run: | ||
working-directory: ./cogstack-es | ||
|
||
jobs: | ||
types-lint-tests: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
matrix: | ||
python-version: [ '3.9', '3.10', '3.11', '3.12' ] | ||
max-parallel: 4 | ||
|
||
steps: | ||
- uses: actions/checkout@v5 | ||
- name: Set up Python ${{ matrix.python-version }} | ||
uses: actions/setup-python@v5 | ||
with: | ||
python-version: ${{ matrix.python-version }} | ||
- name: Install dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install -r requirements.txt | ||
pip install -r requirements-dev.txt | ||
- name: Check types | ||
run: | | ||
python -m mypy --follow-imports=normal *.py | ||
- name: Lint | ||
run: | | ||
ruff check *.py | ||
- name: Test | ||
run: | | ||
pytest tests | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
|
||
# Login and search | ||
This project is responsible for logging in and performing a search. | ||
|
||
## Login details | ||
1. Create a [credentials.py](credentials.py) | ||
2. Populate it with your cogstack instance and login details | ||
An example template can be seen below: | ||
``` | ||
hosts = [] # This is a list of your cogstack elasticsearch instances. | ||
|
||
# These are your login details (either via http_auth or API) | ||
username = None | ||
password = None | ||
``` | ||
|
||
__Note__: If these fields are left blank then the user will be prompted to enter the details themselves. | ||
|
||
If you are unsure about the above information please contact your CogStack system administrator. | ||
|
||
## How to build a Search query | ||
|
||
A core component of cogstack is Elasticsearch which is a search engine built on top of Apache Lucene. | ||
|
||
Lucene has a custom query syntax for querying its indexes (Lucene Query Syntax). This query syntax allows for features such as Keyword matching, Wildcard matching, Regular expression, Proximity matching, Range searches. | ||
|
||
Full documentation for this syntax is available as part of Elasticsearch [query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/query-dsl-query-string-query.html#query-string-syntax). |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
Copilot Autofix
AI about 2 months ago
To fix the issue, add a
permissions
block specifying the least necessary privileges for the job. In this workflow, the job does not push changes, create issues, comment on pull requests, or modify repository content—it only checks out code and runs tests/linting. Therefore, limiting permissions tocontents: read
at the job or workflow root is most appropriate. You can add this block either at the top (outsidejobs:
) so it applies to all jobs, or directly under thetypes-lint-tests:
job section if you only want to affect that job. The best approach is adding to the root so that future jobs inherit safe defaults.Edit
.github/workflows/cogstack-es_main.yml
and insert the block:immediately after the workflow
name:
and before theon:
key.