[16.0][MIG] edi_storage_oca: Migration to 16.0 #26

QuocDuong1306 · 2023-10-10T08:12:06Z

Migrate to v16
Change the dependency from storage_backend to fs_storage

Ack files should be treated as incoming records. Receive files is not handled yet by the framework and will be added soon.

* A cron job check related storage on edi backends to create the needed exchange files. * Add basic receive component.

…all runtime fixes.

…filenames from storage. Also extend help in `exchange_filename_pattern` field to clarify usage.

…nd assigned.

edi_storage_oca/components/base.py

edi_storage_oca/components/check.py

edi_storage_oca/components/listener.py

simahawk · 2023-11-16T14:38:41Z

edi_storage_oca/tests/test_edi_backend_storage.py

@@ -39,7 +39,7 @@ def test_export_file_already_done(self):
        self.record.edi_exchange_state = "output_sent"
        mocked_paths = {self._file_fullpath("done"): self.fakepath}
        # TODO: test send only w/out cron (make sure check works)
-        self._test_run_cron(mocked_paths)
+        self._test_run_cron(mocked_paths, skip_sent=False)


why do we have to add this change?

Because in tests that have some case with the status output_sent so we use the flag skip_sent to ignore another case:

skip_sent = True --> ignore records that were already sent

skip_sent = False ---> send it

I know what it does 😉
I mean: why was not needed before and it is?

edi_storage_oca/components/check.py

edi_storage_oca/components/listener.py

edi_storage_oca/components/receive.py

JordiMForgeFlow · 2023-12-01T08:41:57Z

edi_storage_oca/models/edi_backend.py

+            bits.append(r"\." + exchange_type.exchange_file_ext)
+        pattern = "".join(bits)
+        # TODO: clean this up, .find_files is deprecated in fs_storage
+        full_paths = self.storage_id.find_files(pattern, full_input_dir_pending)


Hi @QuocDuong1306 and @simahawk

I am testing this PR and the method find_files (https://github.com/OCA/storage/blob/1a4d80faaab0ddf6dfc80d867472dd6ce4777c5f/fs_storage/models/fs_storage.py#L373) is not returning the full path from the root for me. Maybe you know what I may be missing.

I have an FTP where the home folder contains /import folder to import files. When the call to the method is done like self.fs.ls(relative_path, detail=False) being relative_path "/import", I just get the filename of the file inside the folder.

I'll have a look soon

@JordiMForgeFlow are you using directory_path? If yes, you might have inconsistent results. See OCA/storage#311

@simahawk I am not using directory_path. Nevertheless, I have been testing and I have been able to get it working fine by assuming that find_files returns only the filename, not the full path.

The problem is mainly that, if for example I have a file test_file.xml in a folder called import, the find_files method returns test_file.xml. Then, when the line return [p[pending_path_len:].strip("/") for p in full_paths] is executed, the filename stored in the exchange record is left as ile.xml (stripping the first characters as it assumes that it is actually the full path)

JordiMForgeFlow · 2023-12-20T20:38:12Z

edi_storage_oca/components/base.py

+            # otherwise is impossible to retrieve input files and acks
+            # (the date will never match)
+            # TODO: clean this up, .get is deprecated in fs_storage
+            return self.storage.get(path.as_posix(), binary=binary)


@simahawk @QuocDuong1306 another issue that I am seeing is when sending a file. In the send component, a check is done before sending to make sure a file is not sent twice. The check ends up in this line, doing a storage.get.

I suppose it has to do with the new fs storage, but doing the get for a file that does not exist raises a different error than FileNotFoundError. The error is:

ValueError: <class 'ftplib.error_perm'>: "550 Can't open *****: No such file or directory" (being ***** the filepath being accessed)

I have created a PR at here: fsspec/filesystem_spec#1494

john-herholz-dt · 2024-01-16T06:21:02Z

What is the stage of this PR?

JordiMForgeFlow · 2024-01-16T07:44:15Z

Hi @john-herholz-dt

It works fine except for the issue I mentioned #26 (comment). I have been using a custom branch where I simply fixed what I commented and it has been working fine for some weeks already.

john-herholz-dt · 2024-01-16T09:22:15Z

@JordiMForgeFlow ,
thanks for clarifying.

Why not add this fix to this branch?
At the moment the code coverage target isn't reached.

Will @simahawk merge it anyway or do we need to get the coverage?
Are you then continuing here or do you need help?

I need this module actually for 17.0 so I am waiting. =)
But thanks for the contribution so far, great work!

JordiMForgeFlow · 2024-01-16T09:33:35Z

@john-herholz-dt

The fix I added is basically changing return [p[pending_path_len:].strip("/") for p in full_paths] for return [p.strip("/") for p in full_paths], as in my case (SFTP connection) I was not getting full paths. However, I am not sure if it has to do with my specific case or if it needs to be like this always.

simahawk · 2024-01-31T09:45:40Z

@john-herholz-dt @JordiMForgeFlow @QuocDuong1306 If it's fine for you all I'd merge what we have here as it is and then we iterate again for fixes (whoever has the chance to do it 1st 😉)

john-herholz-dt · 2024-01-31T09:46:58Z

Super fine!

JordiMForgeFlow · 2024-01-31T19:40:36Z

@simahawk LGTM! I will open a PR with the fix regarding the paths.

simahawk · 2024-01-31T20:12:48Z

/ocabot merge nobump

OCA-git-bot · 2024-01-31T20:12:54Z

Hey, thanks for contributing! Proceeding to merge this for you.
Prepared branch 16.0-ocabot-merge-pr-26-by-simahawk-bump-nobump, awaiting test results.

OCA-git-bot · 2024-01-31T20:18:33Z

Congratulations, your PR was merged at 7abe3a4. Thanks a lot for contributing to OCA. ❤️

etobella · 2024-06-27T16:07:25Z

@QuocDuong1306 why did you change storage_backend to fs_storage if there is no migration scripts there? If we migrate from 15 to 16, all the backends that are related to a storage will fail...

Actually, storage_backend module already exists on 16 and this module should replace the original module. But no migration scrips have been created besides a comment on an issue.

@simahawk @JordiMForgeFlow @lmignon

simahawk · 2024-06-28T09:26:25Z

@nilshamerlinck @QuocDuong1306 could you please have a look and evaluate how to get a script to migrate?
I would assume that storage_backend will still be installed in the old db and that we simply have to create the matching fs.storage records w/ their conf leaving the old storage.backend records untouched.
The conversion part can be a util to be added to fs_storage but let's start small for now 😉

nilshamerlinck · 2024-07-08T08:24:36Z

@simahawk proposal of simple migration script pushed here : #95
note that sensitive parts of conf (host, port, credentials) is coming from the environment and will still need manual adaptation

simahawk and others added 30 commits October 10, 2023 15:10

Add edi_storage

abdf629

edi_storage: add base tests for component

90752eb

edi_storage: fix messages test isolation

d25bceb

[UPD] Update edi_storage.pot

8a374e2

edi_storage 13.0.1.1.0

f63c0e9

edi_storage: adapt to edi api changes

f8ecb62

edi_storage 13.0.1.2.0

fadc537

edi_storage: put ack handling on hold

eadaa9c

Ack files should be treated as incoming records. Receive files is not handled yet by the framework and will be added soon.

edi_storage 13.0.1.3.0

89951ca

edi_storage: adapt to new components lookup

96c6989

edi_storage 13.0.1.4.0

be87c71

[CHG] edi: Use more permissive licence: AGPL-> LGPL

022b812

[UPD] README.rst

bbdbfef

edi_storage 13.0.1.4.1

632ef0c

[FIX] edi_storage: sort key was not working properly

b0cd3f8

edi_storage 13.0.1.5.0

ab2c396

[IMP] edi_storage_oca: isort, black, prettier.

dc6171d

[MIG] Migrate module edi_storage_oca to v14.

b8eb876

[UPD] Update edi_storage_oca.pot

6eda88c

[UPD] README.rst

99ea22d

[IMP] edi_storage_oca: implement basis of the input process.

bc24fd7

* A cron job check related storage on edi backends to create the needed exchange files. * Add basic receive component.

[FIX] edi_oca: show the exchange file name while there is no file yet.

a9e6cf9

edi_storage: improve input process

8c06095

[FIX] edi_storage_oca: support the use or not of filename patern + sm…

c9c1dca

…all runtime fixes.

[IMP] edi_storage_oca: add specific channel and job function definitions

b7b9077

[FIX] edi_storage_oca: support regex pattern searching when fetching …

52a29f7

…filenames from storage. Also extend help in `exchange_filename_pattern` field to clarify usage.

[FIX] edi_storage: consider if the exchange type has a specific backe…

6f7f342

…nd assigned.

[IMP] : black, isort, prettier

2ab5d16

[UPD] Update edi_storage_oca.pot

5d4796d

[UPD] README.rst

4d6e533

simahawk requested changes Nov 16, 2023

View reviewed changes

QuocDuong1306 force-pushed the 16.0-mig-edi_storage_oca branch from 606e358 to 7c88d53 Compare November 20, 2023 04:26

simahawk reviewed Nov 20, 2023

View reviewed changes

edi_storage_oca/components/check.py Outdated Show resolved Hide resolved

QuocDuong1306 force-pushed the 16.0-mig-edi_storage_oca branch from 7c88d53 to d2168b0 Compare November 20, 2023 07:36

QuocDuong1306 requested a review from simahawk November 20, 2023 07:44

QuocDuong1306 force-pushed the 16.0-mig-edi_storage_oca branch from d2168b0 to a2fc4e3 Compare November 20, 2023 08:52

simahawk reviewed Nov 29, 2023

View reviewed changes

edi_storage_oca/components/listener.py Show resolved Hide resolved

edi_storage_oca/components/receive.py Show resolved Hide resolved

[MIG] edi_storage_oca: Migration to 16.0

b0e99b8

QuocDuong1306 force-pushed the 16.0-mig-edi_storage_oca branch from a2fc4e3 to b0e99b8 Compare December 1, 2023 07:37

simahawk approved these changes Dec 1, 2023

View reviewed changes

JordiMForgeFlow reviewed Dec 1, 2023

View reviewed changes

JordiMForgeFlow reviewed Dec 20, 2023

View reviewed changes

JordiMForgeFlow mentioned this pull request Jan 12, 2024

[16.0] fs_storage + EDI: connection closed after some time OCA/storage#319

Closed

OCA-git-bot added the bot is merging ⏳ label Jan 31, 2024

OCA-git-bot merged commit 5256e6b into OCA:16.0 Jan 31, 2024
5 of 7 checks passed

OCA-git-bot added merged 🎉 and removed bot is merging ⏳ labels Jan 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[16.0][MIG] edi_storage_oca: Migration to 16.0 #26

[16.0][MIG] edi_storage_oca: Migration to 16.0 #26

QuocDuong1306 commented Oct 10, 2023

simahawk Nov 16, 2023

QuocDuong1306 Nov 20, 2023

simahawk Nov 29, 2023

JordiMForgeFlow Dec 1, 2023

simahawk Dec 12, 2023

simahawk Dec 18, 2023

JordiMForgeFlow Dec 18, 2023

JordiMForgeFlow Dec 20, 2023

QuocDuong1306 Jan 9, 2024

john-herholz-dt commented Jan 16, 2024

JordiMForgeFlow commented Jan 16, 2024

john-herholz-dt commented Jan 16, 2024

JordiMForgeFlow commented Jan 16, 2024

simahawk commented Jan 31, 2024

john-herholz-dt commented Jan 31, 2024

JordiMForgeFlow commented Jan 31, 2024

simahawk commented Jan 31, 2024

OCA-git-bot commented Jan 31, 2024

OCA-git-bot commented Jan 31, 2024

etobella commented Jun 27, 2024

simahawk commented Jun 28, 2024

nilshamerlinck commented Jul 8, 2024

[16.0][MIG] edi_storage_oca: Migration to 16.0 #26

[16.0][MIG] edi_storage_oca: Migration to 16.0 #26

Conversation

QuocDuong1306 commented Oct 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

john-herholz-dt commented Jan 16, 2024

JordiMForgeFlow commented Jan 16, 2024

john-herholz-dt commented Jan 16, 2024

JordiMForgeFlow commented Jan 16, 2024

simahawk commented Jan 31, 2024

john-herholz-dt commented Jan 31, 2024

JordiMForgeFlow commented Jan 31, 2024

simahawk commented Jan 31, 2024

OCA-git-bot commented Jan 31, 2024

OCA-git-bot commented Jan 31, 2024

etobella commented Jun 27, 2024

simahawk commented Jun 28, 2024

nilshamerlinck commented Jul 8, 2024