change extension in pipeline.yaml when changing scripts/notebooks format #755
Conversation
|
A bit more efficient solution can be reiterating on the changed tasks after change and then reading/writing to the file once, but then we have to maintain the state of the old extensions for changed files. |
| @@ -606,6 +606,9 @@ def format(self, fmt): | |||
| if extension_changed: | |||
| path = self._path.with_suffix(ext_format) | |||
| Path(self._path).unlink() | |||
| modified_entry = (Path(entry_point).read_text().replace( | |||
There was a problem hiding this comment.
there's an edge case here. we also support other types of entry points, for example, it might be a path_to_module.path_to_function string so read_text() will fail in that case.
I suggest checking Path(entry_point).is_file(), if False, we should print a message with click.secho (in yellow font) saying that the entry point is not a file, and skip the extension replacement - the user will have to do it manually.
There was a problem hiding this comment.
Functions doesn't have format so it doesn't even gets to this code, it throws an exception
Out[1]: PythonCallableSource(function) (defined at: '/private/var/folders/qx/px2tf0rn5bq6c79yd3gk37fwzpvmb9/T/tmpgnur3y2z/content/my_tasks/raw/functions.py:4')
'PythonCallableSource' object has no attribute 'format' We can either implement it with the secho or print it as part of the except block
There was a problem hiding this comment.
I didn't mean a function which is a task in a pipeline. I meant an entry point. We support users defining a function that returns a DAG object. Example:
ploomber build --entry-point some_module.some_function
In such case, the code will break because entry_point isn't a file.
Example here: https://docs.ploomber.io/en/latest/user-guide/parametrized.html#Python-API-(factory-functions)
…pipeline_change_624 # Conflicts: # src/ploomber/sources/notebooksource.py
tests/cli/test_nb.py
Outdated
|
|
||
| def test_format_no_entry_point(monkeypatch, tmp_nbs_factory, capsys): | ||
| monkeypatch.setattr(sys, 'argv', [ | ||
| 'ploomber', 'nb', '--entry-point', 'factory.make', '--format', 'ipynb' |
There was a problem hiding this comment.
the CI is failing. I think if you replace factory.make for test_pkg.entry.with_doct it should work since that function is already defined
There was a problem hiding this comment.
I think factory.make isn't working because the module isn't in the path. so if you go with test_pkg you can delete the nbs-factory code and fixture.
edublancas
changed the title
…mat (ploomber#755) * Initial commit * Testing pipeline doesn't change * Fixing tests * minor changes * Added edge tests * lints * Fixing test * Fixing test * fixing test Co-authored-by: Eduardo Blancas Reyes <github@blancas.io>
As part of the inner task extension format, for each change we're editing the pipeline.yaml file