This repository was archived by the owner on Sep 16, 2022. It is now read-only.
rewrite load initial data management command #758
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
katrite
approved these changes
Feb 11, 2021
EarthModule
pushed a commit
that referenced
this pull request
Feb 15, 2021
* rewrite load initial data management command (#758) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * delete flake8 check from travis. (#760) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * fix service role issues (#761) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * Add_metax_env_variable_to_components (#756) Add METAX_ENV to components via version control Co-authored-by: Katri Tegel <katri.tegel@gmail.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> Co-authored-by: Toni Nurmi <tonurmi@csc.fi> Co-authored-by: Katri Tegel <17436908+katrite@users.noreply.github.com> Co-authored-by: Katri Tegel <katri.tegel@gmail.com>
EarthModule
pushed a commit
that referenced
this pull request
Mar 10, 2021
* Bump django from 3.1.2 to 3.1.4 Bumps [django](https://github.com/django/django) from 3.1.2 to 3.1.4. - [Release notes](https://github.com/django/django/releases) - [Commits](django/django@3.1.2...3.1.4) Signed-off-by: dependabot[bot] <support@github.com> * include comparison values in logging when comparing auth password if debug is on * Update-test-db-with-migrations - Add migrations to correspond production db * Add-files-to-att-dataset (#732) Modify single dataset a0651f93-1826-4264-8fe4-df39f1ee72ac in prod Related to changing catalog Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * Add-files-to-att-dataset (#734) Add dataset_version_set to datasets Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * Att test dataset (#736) * Add-files-to-att-dataset -Add test dataset that belongs to IDA-project * att-test-dataset Add att-test-dataset i# with '#' will be ignored, and an empty message aborts the commit. Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * Bump djangorestframework from 3.12.1 to 3.12.2 (#718) Bumps [djangorestframework](https://github.com/encode/django-rest-framework) from 3.12.1 to 3.12.2. - [Release notes](https://github.com/encode/django-rest-framework/releases) - [Commits](encode/django-rest-framework@3.12.1...3.12.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> * Bump lxml from 4.6.1 to 4.6.2 (#733) Bumps [lxml](https://github.com/lxml/lxml) from 4.6.1 to 4.6.2. - [Release notes](https://github.com/lxml/lxml/releases) - [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt) - [Commits](lxml/lxml@lxml-4.6.1...lxml-4.6.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> * Bump pytz from 2020.1 to 2020.5 (#730) Bumps [pytz](https://github.com/stub42/pytz) from 2020.1 to 2020.5. - [Release notes](https://github.com/stub42/pytz/releases) - [Commits](stub42/pytz@release_2020.1...release_2020.5) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> * Merge Test to Stable (#737) * add error handling to migration 27 in case of missing syke.csv file * split settings with django-split-settings, enable root logger, add .env support * replace redis client with new implementation, replace yml file with settings * integrate refdata loading as a task * remove unnecessary 403 error on flush_records * remove the yaml loading blocks * replace configs with env variables * fix some unnecessary asserts in tests * remove unnecessary request logging logic * update .gitignore, deps, and travis. Create .env.template * delete settings.py * update readme * fix ENV_VARS instructions * CSCFAIRADM-375: [ADD] instructions for mac and for db creation (#679) * add watchman system monitoring, bind elasticsearch hosts to env * disable tests testing file permissions, make elasticsearch take its host from env vars, add Docker as an option for local development setup * Increase logging on redis and rabbitmq connections, fix rabbitmq retry behaviour to be faster * optimize imports * CSCFAIRADM-375: [FIX] typos in variables and ignore log directory * CSCFAIRMETA-701: [FIX] crash on v2 dataset rpc api - Prevents creating a new dataset version if there exists a draft * CSCFAIRMETA-701: trigger travis * Update-test-db-with-migrations - Add migrations to correspond production db * Bump django from 3.1.2 to 3.1.4 Bumps [django](https://github.com/django/django) from 3.1.2 to 3.1.4. - [Release notes](https://github.com/django/django/releases) - [Commits](django/django@3.1.2...3.1.4) Signed-off-by: dependabot[bot] <support@github.com> * Modify-JYU-catalog-logo - Remove URL from JYU catalog logo name * Document env vars (#720) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * add OAI env vars OAI_REPOSITORY_NAME, OAI_ETSIN_URL_TEMPLATE and OAI_ADMIN_EMAIL. Add new remote environment (#724) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * Add env var to control if rabbitmq should use vhost. Change RabbitMQ service to use VHOST only if env var allows it (#726) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * CSCFAIRMETA-799-RabbitMQ-TTV (#722) * CSCFAIRMETA-799-RabbitMQ-TTV - Add new connection with new vhost to TTV - Add new exchange and routing_key for TTV - Publish messages to 1 queue for TTV * CSCFAIRMETA-799-RabbitMQ-TTV - Add rabbitmq consumer for ttv * CSCFAIRMETA-799: [FIX] simplify publish process - Use the same virtual host for ttv and etsin - Adding exchanges should now only need to define it in settings - Define queues in settings - Move rabbitmq consumers to local environment since they are not used in other places Co-authored-by: Katri Tegel <katri.tegel@gmail.com> Co-authored-by: Tommi Pulli <tommi.pulli7@gmail.com> * CSCFAIRMETA-799-RabbitMQ-TTV-fix-settings (#728) -Rename django_settings to settings Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * Add-files-to-att-dataset (#732) Modify single dataset a0651f93-1826-4264-8fe4-df39f1ee72ac in prod Related to changing catalog Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * Add ENVs to travis.yml and fixed related settings - added SSL for elasticsearch in ENV - rabbitmq vhost is name is not needed if it's not used - travis environment settings are the same as in test environment * linting and removed unnecessary setting variable * fix flake8 problems and ones that are consequent - api_access settings were defined too permissive for unit tests - uncommented previously commented auth tests because of the api_access change - deleted a file permission related test that tests trivial issue and causes a headache in the new docker local env * add directory for logs during travis before_script * add rabbitmq hosts to default env * start using real redis in travis - Ignore console logging in Travis - test/stable deploys were removed since they aren't used anyways * Add-files-to-att-dataset (#734) Add dataset_version_set to datasets Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * Att test dataset (#736) * Add-files-to-att-dataset -Add test dataset that belongs to IDA-project Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * Cscfairmeta 847 relax draft validation (#740) * CSCFAIRMETA-847: [ADD] needed changes in draft json schema * CSCFAIRMETA-847: [FIX] draft datasets use dft schema - Add extra validation to publishing method to validate against the given data catalog * CSCFAIRMETA-847: [ADD|FIX] tests * CSCFAIRMETA-890-DOI-as-pref-identifier (#741) * CSCFAIRMETA-890-DOI-as-pref-identifier - Add a check for DOIs from remote datasets - Add a check that remote datasets with DOI are validated against datacite * CSCFAIRMETA-890-DOI-as-pref-identifier - Modify check for which DOIs are sent for registration Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * CSCFAIRMETA-776: [ADD] Bulk delete (#725) * CSCFAIRMETA-776: [ADD] Bulk delete * CSCFAIRMETA-776-bulk-delete -Enable deleting of drafts Co-authored-by: genie9 <evgenia.lyjina@codento.com> Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * CSCFAIRMETA-847: [FIX] bug where files could not be added to drafts * Api endpoints env (#743) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * remove test comparing datetimes down to seconds as not needed (#744) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * Bump djangorestframework from 3.12.1 to 3.12.2 (#718) Bumps [djangorestframework](https://github.com/encode/django-rest-framework) from 3.12.1 to 3.12.2. - [Release notes](https://github.com/encode/django-rest-framework/releases) - [Commits](encode/django-rest-framework@3.12.1...3.12.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> * Bump lxml from 4.6.1 to 4.6.2 (#733) Bumps [lxml](https://github.com/lxml/lxml) from 4.6.1 to 4.6.2. - [Release notes](https://github.com/lxml/lxml/releases) - [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt) - [Commits](lxml/lxml@lxml-4.6.1...lxml-4.6.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> * Bump pytz from 2020.1 to 2020.5 (#730) Bumps [pytz](https://github.com/stub42/pytz) from 2020.1 to 2020.5. - [Release notes](https://github.com/stub42/pytz/releases) - [Commits](stub42/pytz@release_2020.1...release_2020.5) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> Co-authored-by: Tommi Pulli <43133956+tompulli@users.noreply.github.com> Co-authored-by: Tommi Pulli <tommi.pulli7@gmail.com> Co-authored-by: Katri Tegel <katri.tegel@gmail.com> Co-authored-by: Toni Nurmi <tonurmi@csc.fi> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Evgenia Lyjina <jenny.lyjina@gmail.com> Co-authored-by: genie9 <evgenia.lyjina@codento.com> * fix near production api access scheme (#747) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * add migration to fix incorrect file counts on directories (#752) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * hotfixes to stable before production update (#748) * fix near production api access scheme (#747) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * add migration to fix incorrect file counts on directories (#752) Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * make new management command to verify filecount for directories (#755) * make new management command to verify filecount for directories * fix/directory_file Fix typos s Co-authored-by: Toni Nurmi <tonurmi@csc.fi> Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * rewrite load initial data management command (#758) * rewrite load initial data management command Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * delete flake8 check from travis. (#760) * delete flake8 check from travis. * cd into src before calling manage.py Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * fix service role issues (#761) * fix service role issues * add draft perms Co-authored-by: Toni Nurmi <tonurmi@csc.fi> * Add_metax_env_variable_to_components (#756) Add METAX_ENV to components via version control Co-authored-by: Katri Tegel <katri.tegel@gmail.com> Co-authored-by: Toni Nurmi <toni.nurmi@csc.fi> * CSCFAIRMETA-912-recalculate-bytesize-filecount-for-cr (#765) - Recalculate byte_size and file_count for catalog record Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * CSCFAIRMETA-954-publish-records-to-TTV (#767) - Fetch and publish all existing datasets into, and only to, TTV-operations queue Co-authored-by: Katri Tegel <katri.tegel@gmail.com> * include comparison values in logging when comparing auth password if debug is on * clear API_USERS setting if server is restarted Co-authored-by: Katri Tegel <17436908+katrite@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Toni Nurmi <tonurmi@csc.fi> Co-authored-by: Katri Tegel <katri.tegel@gmail.com> Co-authored-by: Tommi Pulli <43133956+tompulli@users.noreply.github.com> Co-authored-by: Tommi Pulli <tommi.pulli7@gmail.com> Co-authored-by: Evgenia Lyjina <jenny.lyjina@gmail.com> Co-authored-by: genie9 <evgenia.lyjina@codento.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.