Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge nested env keys when using import_from #1034

Merged
merged 1 commit into from Oct 14, 2022
Merged

Merge nested env keys when using import_from #1034

merged 1 commit into from Oct 14, 2022

Conversation

judahrand
Copy link
Contributor

@judahrand judahrand commented Oct 13, 2022 •
edited

Describe your changes

Previously, if you had nested keys in your env.yaml it wasn't possible to override a subset of them.

For example:

env.yaml

nested:
  key_1: value_1
  key_2: value_2

and
env.prod.yaml

meta:
  import_from: env.yaml

nested:
  key_1: new_value_1

would result in:

nested:
  key_1: new_value_1

With these changes the result is:

nested:
  key_1: new_value_1
  key_2: value_2

This should work to an arbitrary depth.

Issue ticket number and link

Closes #x

Checklist before requesting a review

  • I have performed a self-review of my code
  • I have added thorough tests (when necessary).
  • I have added the right documentation (when needed). Product update? If yes, write one line about this update.
    • I don't think this really needs any additional docs as this is the expected behaviour I'd have thought?

@edublancas edublancas merged commit 139b21f into ploomber:master Oct 14, 2022
@edublancas
Copy link
Contributor

this is great, thanks for your contribution!

@judahrand judahrand deleted the feature/env-merge branch October 14, 2022 19:49
neelasha23 pushed a commit to neelasha23/ploomber that referenced this pull request Jan 5, 2023
@judahrand @neelasha23
Merge nested env keys when using import_from (ploomber#1034)
1c6d173
idomic added a commit that referenced this pull request Jan 12, 2023
@neelasha23 @Wxl19980214
@e1ha @edublancas @94rain @qixuan27 @yafimvo @yafim @idomic @grnnja @elongl @dependabot @balapriyac @boscacci @amdflake @shizuchanw @AnInventor @pelincetin @judahrand @mhered @Prathamesh010 @Mustapha-AJEGHRIR @Nekros1712 @RodolfoFerro @bstee615 @sumanpaikdev @feribg @WSShawn
[Docs] File client user guide (#1060)
f5a2f0e 
* remove xfail test and add to ci integration test

* remove email onboarding test (#914)

remove email onboarding test

* documents how to index snippets on algolia (#915)

* Added an Algolia Crawler Section to doc

* Update contributing.md

* allow dotted paths in grid spec

* allows TaskGroup to pass a placeholder in the namer arg

* closes #698

* documents task name placeholders in grid

* cache dag in jupyter plugin - closes #894

* updates changelog

* ploomber release 0.19.8

* Bumps up ploomber to version 0.19.9dev

* Added D3 multiple backend support (#904)

* add --backend to report command

* update --backend help msg

* Add initial report command tests

* add backend arg to dag.plot

* try using `plot.choose_backend(backend)`

* add `dot -c` to win CIs

* update help msg and remove backend in cli msg

* remove backend assertion

* address feedback

* add did you mean feature to `ploomber examples` closes #805 (#825)

* added and tested issue #805

* fix tepo issues based on Ido's feedback

* pass test

* add psycopg2-binary back

* fix bugs to pass test

* delete unused files

* deleted fileclient/.gitignore

* removed the commented debug

* deleted the blank line at end of file

* no message

* fixed bugs with format checked

* add one test

* checked format with flake8

* removed one test

* fixed mock called twice error

* fixed test_error_unknown_example

* fixed format issues

* solved format issues

* fixed format issues

* fixed CI errors

* removed debug code

* avoid creation of multiple example managers and add more test cases

* add `_suggest_example` and revert arg order change

* refactor: move file read to constructor

* use try-except to bypass exceptions

* print error instead of raising exception

* Update newline breaks

Co-authored-by: shuyang <94rain@msn.com>
Co-authored-by: shuyang <21193371+94rain@users.noreply.github.com>

* add custom papermill engine for debugging - closes #823

* updates changelog

* fixes typo in docs

* Handle exceptions in `jupyter.manager.load_dag` (#923)

* handle exception in `jupyter.manager.load_dag`

* Add `test_load_dag_exception_handling` test

* ensure `log.exception.call_args_list` is expected

* fixes typo in docs - closes #933

* Fixed confusing error when passing something.html as source #646 (#920)

* Fixed confusing error when passing something.html as source #646

* Failing test fixed

* Error message fixed

* _looks_like_file_name check added after import fails

* typo fixed. test fixed.

Co-authored-by: yafim <ifim.vo@gmail.com>

* Enabling GPU context through start_method for parallel executor (#927)

* start_method added to parallel executor

* tests added to executor with valid start_method values

* updates contributing.md

* updates contributing.md

* fixes markdown heading

* style improvements

* fixes error in contributing.md

* Add 3.6 deprecation warning to ploomber and ploomber_cli (#932)

* Add Python 3.6 deprecation warning

* temp downward to 3.6

* Add deprecation warning to CLI as well

* restore nbclient req and _PY_DEFAULT_VERSION

* update deprecation warning message

* update deprecation warning message

* Update __init__.py

* Update __init__.py

Co-authored-by: Ido M <michael.ido@gmail.com>

* disables custom papermill engine

* updates changelog

* deletes notebook from dependencies

* updates changelog

* adds cmd shortcuts to tasks.py

* ploomber release 0.19.9

* Bumps up ploomber to version 0.19.10dev

* Fix d3 report missing plot output (#946)

* fix d3 report output

* Fix d3 output when no plot is required

* Remove Python 3.6 deprecation warning (#947)

* Dropping support for python 3.6 (#876)

* Updated the telemetry's documentation. (#948)

Argo's comment was incorrect, referred to Airflow.

* capture exceptions when dotted path in grid fails - closes #951

* allows using previous values when resolving env

* flake8 fixes due to 5.0 update

* updates changelog

* adds test for _jupytext_fmt function

* unit tests for onboarding tutorial (#949)

* adding unit tests for onboarding tutorial

* add few unit tests

* mock email_input

* mock input function as well

* mock in onboard

* adding a return val for magic_mock

* try skip python 3.7

* doc updates ok-to-test and dev/ branch + CI improvements (#955)

added doc in contribute.md

fixed typo

switch location of branch name doc

switch location of branch name doc

added link to ok-to-test

* unused env.yaml placeholders warning takes ignores inline placeholders

* Update README.md

* Adding missing jupyter links on ploomber installation (#963)

* doc: page title prefixes section name

* first implementation of --debuglater feature

* adds function to validate input arguments

* debug renamed to debug_mode to prevent conflicts with debug method

* minor changes to PythonCallable

* updates tests

* updates "ploomber task" due to debug -> debug_name change

* no longer using debug_if_exception in DAG

* minor changes to NotebookRunner

* adds kwargs to debug_if_exception

* moving debug_if_exception to PythonCallable

* gracefully exiting debugging sessions

* docstring updates

* test updates

* updates cookbook/debugging

* adds note on --debulater dump size

* adds note on dill and serialization

* changing debug_mode from True to "now"

* do not start debugger if user has a breakpoint

* fixing windows test

* clearing screen before updating status (parallel executor)

* changelog updates

* ploomber release 0.20

* Bumps up ploomber to version 0.20.1dev

* Docs fix - copy on click (#962)

* updating Jinja2 and MarkupSafe versions (#970)

* Adding binder button to readme (#969)

* adds progress bar to parallel executor

* grid requires strict dotted path format (a.b::c)

* allows dotted paths in params - closes #477

* raising errors on wrong type or non returned in dotted paths in params and grid

* fixes error due to change in papermill 2.4.0

* doc update: add info on locally running github actions (#977)

* updating mistune and nbconvert dependencies (#976)

* removes telemetry logic, migrates to use ploomber-core (#960)

* remove telemetry related code

add package name to log_api

added version

* setup add ploomber core

* fixed tests

* fixed test error

* lint fixed

* changed posthog api key to a parameter

* fixed bug

* use telemetry as class object

* posthog project key added

* lint fixed

* fixes top-level import

* creating telemetry instance in one place

* flake8

* only create one instance of telemetry and import it elsewhere

Co-authored-by: Eduardo Blancas Reyes <github@blancas.io>

* fixes tests compatibility with ipython 8 (#978)

* added mocked interact class

* ipython<8 limitation removed

* Update setup.py

Co-authored-by: yafim <ifim.vo@gmail.com>
Co-authored-by: Eduardo Blancas <edu.blancas@gmail.com>

* deletes old python 3.6 dependency

* removes restriction on jinja2 version

* 882 switch start method in parallel executor (#957)

* start_method added to parallel executor

* tests added to executor with valid start_method values

* documentation added. default start_method test added. code fixed.

* version added to documentation

* Update CHANGELOG.md

* example added to spec.rst

Co-authored-by: yafim <ifim.vo@gmail.com>

* clean traceback when using ploomber task (#889)

* clean traceback when using ploomber task (#828)

* some fixes

* making task exceptions shorter

* add clean traceback to changelog and add summary toggle

* add test util function to set terminal width

Co-authored-by: Eduardo Blancas Reyes <github@blancas.io>

* moved logic to add parameters cell (when missing) to the Python API (#971)

* Fix unintended side effect

* fix extract_up and extract_prod

* remove redundant args

* Address some of the PR feedback

* move _validate_parameters_cell to NotebookSource

* remove _post_init_validation body and add add_parameters_cell condition

* remove tests that are no longer applicable

* Update source addition and use numpydoc

* Add `add_parameters_cell_if_missing` tests for NotebookRunner and ScriptRunner

* Fix typo

* update CHANGELOG

* fixes empty notebook edge case

* fix

Co-authored-by: Eduardo Blancas Reyes <github@blancas.io>

* compatibility with mistune 0.8.x and 2.x

* adds ParallelDill executor

* fixes type checking in InMemoryDAG

* support for functions whose arguments do not match function names

* adds grid decorator

* capture decorator

* adds progress to ParallelDill

* adds callback error to dill executor

* adds missing testing dependency

* renames ploomber.inline -> ploomber.micro

* fixes problem when using _Client

* adds deprecation warning to {NotebookRunner, PythonCallable}.develop()

* adds hot_reload test in micro pipelines

* updates changelog

* adds boilerplate for pair decorator

* text fixes, fix dag.plot

* updates changelog

* adds timeout to unit tests

* fixing timeout

* skipping some tests that are getting stuck

* ploomber release 0.21

* Bumps up ploomber to version 0.21.1dev

* updates binder links

* patching test

* Bump nbconvert from 6.3.0 to 6.5.1 (#992)

Bumps [nbconvert](https://github.com/jupyter/nbconvert) from 6.3.0 to 6.5.1.
- [Release notes](https://github.com/jupyter/nbconvert/releases)
- [Commits](jupyter/nbconvert@6.3.0...6.5.1)

---
updated-dependencies:
- dependency-name: nbconvert
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* change Get Started button link to binder (#993)

* fixes telemetry instantiation

* flake8

* 902 cli cell injection with priority (#985)

* cell_injection_test

* cli cell injection with priority

* cli cell injection with priority

* --priority args added to nb --inject to set which parameters to inject

* keyerror fixed

* nb_inject_test assets location updated

* warning message fixed

* pipeline.yaml location changed

* inject_default_task_when_no_priority_given fixed

* inject_default_task_when_no_priority_given removed

* changelog updated

* test re-pushed

* get_assets_for_testing_inject method added

* flake8

Co-authored-by: yafim <ifim.vo@gmail.com>

* Revert "902 cli cell injection with priority (#985)" (#999)

This reverts commit 6a41fc8.

* Docs update:  basic-concepts.rst (#990)

* Update basic-concepts.rst

- Fix an inadvertent typo (redundant out towards the end)
- Introduce sentence breaks for easier parsing
- For example, .<content> reads better than (eg., <content>)
- See <this-guide> reads better than Click here (updated one instance)

* Update basic-concepts.rst

* add intro tutorial (#1002)

* removing onboarding cli workflow (#996)

* updates cookbook/grid to use placeholders (#1001)

* Added parameter placeholders

* Changed the comments

* fixes error when pairing notebooks (#979)

* fix inject cell error after pairing notebooks

* fixed bug 797

* added moved scripts test file

* changed test to use nb-nested

* fixes typo (#1005)

* deletes unused config module (moved to the core pkg)

* Fixing the cloud download function (#1010)

* Fixing file download function

* lints

* Adjusted api._download_file function (#1012)

* Adjusted function

* removing imports

* dev/doc_contributing_1007: Added fork instruction and image for guidance (#1018)

* Added fork image

* Modified fork.png

* Added the fork image and respective instruction.

Co-authored-by: Hari Bala <HariBala@Nowhere.comm>

* adds cloud task command

* adds "ploomber cloud abort @latest" and --json arg to some cloud cmds

* fixes error in commander

* updates changelog

* documents how to exit DAG gracefullt via special exception (#1004)

* Update serial.py

* updated docs

* checked formatting

* updated docs and checked for linting

* removed comments for Linux tests

* fixed error for linux

* rerun after IRKernel installation error

* updated doc examples

* updated early_stop_rot

* linting

* Cli hiding traceback (#1003)

* Update io.py

* Update io.py

* checked format and linting

* added test for command_endpoint

* re-run test

I saw that CI in the morning failed because of NetworkExceptions; it passed my local tests so I'm committing this change to rerun the test

* Update test_examples.py

* Update test_examples.py

* switch embedded plot to use IFrame (#1017)

* Change IPython display to use IFrame

* Change the styling

* Embedded file path fixed

* embedded-assets dir fixed

Co-authored-by: pelincetin <pc2807@columbia.edu>
Co-authored-by: yafim <ifim.vo@gmail.com>

* fixes #1024

* allow using @ latest in log and status cloud commands

* updates trigger and mark-failed endpoints to take a runid

* adds --json arg when listing cloud runs, updates /trigger endpoint

* cloud allows environment.lock.yml

* updates changelog

* ploomber release 0.21.1

* Bumps up ploomber to version 0.21.2dev

* Allow for setting `null` clients (#1025)

* abstracts params cell parsing logic in ParamsCell

* parsing default values in noteook parameters cell, adds tests

* adds options to upload/download data at specific locations

* wrapping text in D3 plot for tasks with long names (#968)

* Make font size dynamic

* Wrap text in circle

* Added foreignobject tag and scrollbar to wrap text

* Centered text and moved text up

* Made font size arial

* Text position and length fixed

* text wrapper added

Co-authored-by: yafim <ifim.vo@gmail.com>

* Right bar in docs (#1031)

* Created new button for side bar

* Right bar display on mobile fixed

* Right bar display on mobile fixed

* font-size changed to rem

* .gitignore restored

Co-authored-by: e1ha <haelaine6@gmail.com>
Co-authored-by: grnnja <grnnja@gmail.com>
Co-authored-by: yafim <ifim.vo@gmail.com>

* Note on naming of product dict keys and fix typos (#1033)

* Add a note to clarify that the keys of product dictionaries can
be given any name, in files  doc/get-started/basic-concepts.rst
and doc/api/spec.rst

* Fixing minor typos in both files.

* Merge nested `env` keys when using `import_from` (#1034)

* Fix #773 (#1026)

* added param example on documentation (#1029)

* added param example on documentation

* changed env vars into placeholders

Co-authored-by: Mustapha AJEGHRIR <mustapha.ajeghrir@kleegroup.com>

* refactors cloud api

* updates changelog

* ploomber release 0.21.2

* Bumps up ploomber to version 0.21.3dev

* Added a condition to check existence of section (#1039)

* fixes call to cli

* ploomber release 0.21.3

* Bumps up ploomber to version 0.21.4dev

* cli cloud improvements

* printing sign up url to get api key when missing

* ploomber cloud nb accepts urls as argument

* test fixes

* removes unnecessary stuff from integration tests

* fix

* pins pytest 7.1.* and dill to fix ci

* updates changelog

* ploomber release 0.21.4

* Bumps up ploomber to version 0.21.5dev

* Adds hooks to cloud notebooks in projects (#1041)

* do not copy dag clients

* adds internal method that does not copy the dag when building partially

* fixes DAG.execute_partially

* fixes error when downloading products and metadata is missing

* ploomber release 0.21.5

* Bumps up ploomber to version 0.21.6dev

* ci

* updates cloud index

* Add improved error message when missing product key (#1027)

* Add helpful example to validation error message when error happens inside yaml validation

* Handle case where the spec dictates notebook output

* Move ValidationError to DAGSpec from TaskSpec

* Dump JSON when input to DAGSpec is a dictionary

* Correctly check whether data argument is a filepath or dict

* Fix formatting and dangling import

* Fix for flake8

* Fix string concatenation

* Add unit tests for validation help message

* Fix flake8 errors

* Fix typo in unit test expected output

* Remove backslash to appease flake8

* Run formatter

* clearer output when deleting cloud products

* adds --summary option to cloud status command

* adds --summary to the cloud download command

* updates changelog

* ploomber release 0.21.6

* Bumps up ploomber to version 0.21.7dev

* Update ARCHITECTURE.md (#1043)

Missing full stop.

* Prioritized cell injection (#1019)

* cell_injection_test

* cli cell injection with priority

* cli cell injection with priority

* --priority args added to nb --inject to set which parameters to inject

* keyerror fixed

* nb_inject_test assets location updated

* warning message fixed

* pipeline.yaml location changed

* inject_default_task_when_no_priority_given fixed

* inject_default_task_when_no_priority_given removed

* changelog updated

* test re-pushed

* get_assets_for_testing_inject method added

* flake8

* prioritized cell injection

* changelog

* test_inject_remove fixed

* test fixed

* documentation added

Co-authored-by: yafim <ifim.vo@gmail.com>

* Adding stats to cloud commands.

* revert

* Adding cloud stats (#1044)

* Adding stats

* Removing payload

* Removing decorators

* Removing decorators

* lints

* Fixing cmd tracking

* ploomber release 0.21.7

* Bumps up ploomber to version 0.21.8dev

* Add environment vars expansion and pre defined git hashes in the yaml… (#1042)

* Add environment vars expansion and pre defined git hashes in the yaml file

* Add documentation and fix tests.

* Fix flake8 lint

* adds support for zipped notebooks

* Updates cloud notebooks sections (#1047)

* Adds update to cloud notebooks.

* Updates hooks for pipelines api

* Updates pipelines api user guide

* Update CONTRIBUTING.md

* adds note on code migration to ploomber-core

* fixes CI, temporarily removes Python 3.7 from CI

* fixes CI, drops Pyhon 3.7 CI

* updates changelog

* ploomber release 0.21.8

* Bumps up ploomber to version 0.21.9dev

* build-docs

* deletes reference to deprecated jupyterhub

* fixes some broken links

* adds cron job to look for broken links

* inject only NotebookRunner tasks (#1056)

Co-authored-by: yafim <ifim.vo@gmail.com>

* Update License (#1061)

Update License to --> Copyright 2022-Present Ploomber Inc.

* ci hotfix

* updates changelog

* ploomber release 0.21.9

* Bumps up ploomber to version 0.21.10dev

* Added doc

* Revert changes

* changelog

* license

* minor change

* review

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Xilin Wang <stinsionwang@gmail.com>
Co-authored-by: Xilin <35501989+Wxl19980214@users.noreply.github.com>
Co-authored-by: e1ha <59583520+e1ha@users.noreply.github.com>
Co-authored-by: Eduardo Blancas Reyes <github@blancas.io>
Co-authored-by: shuyang <21193371+94rain@users.noreply.github.com>
Co-authored-by: qixuanliu <75308770+qixuan27@users.noreply.github.com>
Co-authored-by: shuyang <94rain@msn.com>
Co-authored-by: yafimvo <103026689+yafimvo@users.noreply.github.com>
Co-authored-by: yafim <ifim.vo@gmail.com>
Co-authored-by: Ido M <michael.ido@gmail.com>
Co-authored-by: Prem Netsuwan <grnnja@gmail.com>
Co-authored-by: Elon Gliksberg <elongliks@gmail.com>
Co-authored-by: Eduardo Blancas <edu.blancas@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bala Priya C <47279635+balapriyac@users.noreply.github.com>
Co-authored-by: Robert Boscacci <cinemarob1@gmail.com>
Co-authored-by: amdflake <62575483+amdflake@users.noreply.github.com>
Co-authored-by: Flora Wang <50822201+shizuchanw@users.noreply.github.com>
Co-authored-by: AnInventor <80908167+AnInventor@users.noreply.github.com>
Co-authored-by: Hari Bala <HariBala@Nowhere.comm>
Co-authored-by: pelincetin <pc2807@columbia.edu>
Co-authored-by: Judah Rand <17158624+judahrand@users.noreply.github.com>
Co-authored-by: e1ha <haelaine6@gmail.com>
Co-authored-by: Manuel Heredia <75589082+mhered@users.noreply.github.com>
Co-authored-by: Prathamesh Gawas <prathameshgawas87@gmail.com>
Co-authored-by: mustapha ajeghrir <66799406+Mustapha-AJEGHRIR@users.noreply.github.com>
Co-authored-by: Mustapha AJEGHRIR <mustapha.ajeghrir@kleegroup.com>
Co-authored-by: Samprit JC <sampritchaurasiya07@gmail.com>
Co-authored-by: Rodolfo Ferro <rodolfoferroperez@gmail.com>
Co-authored-by: Benjamin Steenhoek <benjaminjsteenhoek@gmail.com>
Co-authored-by: Suman Paik <sumanpaik.dev@gmail.com>
Co-authored-by: Feras <feribg@gmail.com>
Co-authored-by: WSShawn <52613022+WSShawn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edublancas edublancas edublancas approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@judahrand @edublancas