Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data Importer - Date Range Issues #5980

Closed
ballj opened this issue Apr 9, 2022 · 8 comments
Closed

Data Importer - Date Range Issues #5980

ballj opened this issue Apr 9, 2022 · 8 comments
Labels
bug Verified and replicated bugs and issues. fixed Bugs that are fixed (in a coming release).

Comments

@ballj
Copy link

ballj commented Apr 9, 2022 •
edited

Description

When using the data importer, I configured it to only download from Nordigen for the last 7 days. The download worked as expected, and I saved the config file.

The config file looks to have hardcoded the following values:

    "date_range": "partial",
    "date_range_number": 7,
    "date_range_unit": "d",
    "date_not_before": "2022-03-27",
    "date_not_after": "",

When running the import again a few days later, it only imported from the 27th onwards, rather than the last 7 days. Any transactions I delete were then re-added.

I then removed the line:

    "date_not_before": "2022-03-27",
    "date_not_after": "",

The remaining config was then:

    "date_range": "partial",
    "date_range_number": 7,
    "date_range_unit": "d",

This did not seem to work and imported as many transactions as Nordigen sent - roughtly a few months.

Expected behaviour

Using the following config I would expect only the last ~7d to be imported:

    "date_range": "partial",
    "date_range_number": 7,
    "date_range_unit": "d",

Debug information

Will have to re-run to provide. Please let me know if this is required.

Steps to reproduce

  1. Configure the data import to use date range
  2. Import a data set larger than the range
  3. Check to see if the whole data set imported

Additional info

No response
@JC5
Copy link
Member

JC5 commented Apr 9, 2022

Which version of the data importer are you using?

@JC5
Copy link
Member

JC5 commented Apr 9, 2022

I am curious to see some logs, because despite the hard-coded value the date is recalculated, which is something that your logs will show:

[2022-04-09 21:01:04] production.DEBUG: Range is partial, after is NULL, dateNotBefore will be calculated.
[2022-04-09 21:01:04] production.DEBUG: dateNotBefore is now "2022-04-06"

Also, the import shows you if and why it is filtering transactions downloaded from Nordigen:

[2022-04-09 21:01:12] production.DEBUG: Skip transaction because "2022-01-21 00:00:00" is before "2022-04-06 00:00:00".

Any transactions I delete were then re-added.

If you delete transactions from Firefly III, they will not be re-imported unless you turn off the duplicate detector.

@JC5 JC5 added the triage Still checking this out... label Apr 9, 2022
@ballj
Copy link
Author

ballj commented Apr 9, 2022

Using verion 0.9.3 of the data importer.
Just filtering out the full logs now. Cant find any line that references the range filter.

Here is part of them:

[2022-04-09 21:21:32] production.INFO: Going to filter downloaded transactions. Original set length is 37
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-04-06 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-04-06 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-04-05 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-31 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-28 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-28 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-24 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-24 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-21 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-21 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-17 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-17 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-15 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-15 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-14 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-14 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-03 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-02 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-02 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-03-01 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-02-28 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-02-28 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-02-21 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-02-15 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-02-09 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-02-03 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-02-03 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-31 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-31 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-28 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-25 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-24 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-24 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-21 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-20 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-19 00:00:00".
[2022-04-09 21:21:32] production.DEBUG: Returning book date
[2022-04-09 21:21:32] production.DEBUG: Include transaction because date is "2022-01-18 00:00:00".
[2022-04-09 21:21:32] production.INFO: After filtering, set is 37 transaction(s)

The config:

{
    "version": 3,
    "source": "fidi-0.9.0",
    "created_at": "2022-03-24T14:33:03+01:00",
    "date": "",
    "default_account": 4,
    "delimiter": "comma",
    "headers": false,
    "rules": true,
    "skip_form": true,
    "add_import_tag": false,
    "roles": [],
    "do_mapping": [],
    "mapping": [],
    "duplicate_detection_method": "cell",
    "ignore_duplicate_lines": false,
    "ignore_duplicate_transactions": false,
    "unique_column_index": 0,
    "unique_column_type": "external-id",
    "flow": "nordigen",
    "identifier": "0",
    "connection": "0",
    "ignore_spectre_categories": false,
    "map_all_data": false,
    "accounts": {
        "XXXXXXXXXXXXXXXXXXXXXXXXX": X,
        "XXXXXXXXXXXXXXXXXXXX": X
    },
    "date_range": "partial",
    "date_range_number": 7,
    "date_range_unit": "d",
    "nordigen_country": "GB",
    "nordigen_bank": "XXXXXXXXXXXXXXXXXXXXXX",
    "nordigen_requisitions": {
        "XXXXXXXXXXXXXXXXXXXX": "XXXXXXXXXXXXXXXXXXXXXXXXXX"
    },
    "conversion": false
}

@JC5
Copy link
Member

JC5 commented Apr 9, 2022

How do you import data? Are you using the UI, the POST command, something else?

@ballj
Copy link
Author

ballj commented Apr 9, 2022

Using the UI -> Import from Norgigen.

Do you need the rest of the logs?

@JC5
Copy link
Member

JC5 commented Apr 10, 2022 •
edited

Yes please.

When I upload a similar configuration through the UI, I see the following lines in my logs.

[2022-04-10 05:45:47] local.DEBUG: Now in updateDateRange()  
[2022-04-10 05:45:47] local.DEBUG: Range is partial, after is NULL, dateNotBefore will be calculated.  
[2022-04-10 05:45:47] local.DEBUG: dateNotBefore is now "2022-03-11"

This is correct eevn though I added a bad date on purpose:

    "date_range": "partial",
    "date_range_number": 30,
    "date_range_unit": "d",
    "date_not_before": "2022-04-08",

Further ahead, I see this:

[2022-04-10 05:47:04] local.INFO: Going to filter downloaded transactions. Original set length is 294  
[2022-04-10 05:47:04] local.INFO: Will not grab transactions before "2022-03-11 00:00:00"

So for me, this works as expected.

@JC5 JC5 added wontfix and removed triage Still checking this out... labels Apr 12, 2022
@ballj
Copy link
Author

ballj commented Apr 13, 2022

Sorry for the delay, logs can be found at https://pastebin.com/RC9C8AQ6

JC5 added a commit to firefly-iii/data-importer that referenced this issue Apr 14, 2022
@JC5
Make sure date range is updated. firefly-iii/firefly-iii#5980
497ec7f
@JC5
Copy link
Member

JC5 commented Apr 14, 2022

No problem! I think I found the issue, and the latest develop version should fix it.

@JC5 JC5 added bug Verified and replicated bugs and issues. fixed Bugs that are fixed (in a coming release). and removed wontfix labels Apr 14, 2022
@JC5 JC5 closed this as completed May 5, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Verified and replicated bugs and issues. fixed Bugs that are fixed (in a coming release).
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JC5 @ballj