Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handles inclusion of Remote Path in FTP filename #948

Closed
wants to merge 1 commit into from

Conversation

jermnelson
Copy link
Collaborator

Fixes error in #944's comment.

Will want @ahafele to maybe test in either dev or stage first before merging and deploying to production just in case there are additional issues in other parts of either the VMA or data-export.

@ahafele
Copy link

ahafele commented May 1, 2024

I tested this for a couple of interfaces on stage. For some interfaces (like the one that prompted this report) the remote path name now needs to be included in the regex, for others they are working as-is. How can we instruct staff to know when to include the remote path in the regex for file fetching - for existing and future interfaces?

@ahafele
Copy link

ahafele commented May 2, 2024

We have another example of an interface that has stopped fetching files - https://sul-libsys-airflow-prod.stanford.edu/vendor_management/interfaces/45. I am also wondering how the proposed fix might impact files that have already been retrieved. If we change the regular expression will previously fetched files be fetched again since they will have a new name.

@ahafele
Copy link

ahafele commented May 2, 2024

I tested the new vendor that was reported today and it appears to be the same issue. When I update the regex to include the pathname then the files are retrieved - https://sul-libsys-airflow-stage.stanford.edu/vendor_management/interfaces/145

Though the files are retrieved they are not being processed and I don't see in the DAGs any info about why (no errors etc.) You can see in the test from yesterday that the files have not been processed and have a status of loading
https://sul-libsys-airflow-stage.stanford.edu/vendor_management/interfaces/139

@jermnelson
Copy link
Collaborator Author

Keeping this PR around until this PR #957 is tested and reviewed to preserve the comments.

@jermnelson
Copy link
Collaborator Author

closing in favor of merged PR #957

@jermnelson jermnelson closed this May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants