fixed broken leumi card adapter #219

liranay · 2019-07-12T06:32:47Z

LeumiCard changed their implementation - they now use REST calls for each search in transactions page.
The change uses puppeteer to get into leumi card website, and then use REST to get the transaction data for each month (it was scraping the UI before, which now does not exists).

…fetch transactions

esakal

@liranso great job!
I left a comment regarding pending status.

esakal · 2019-07-12T13:52:13Z

src/scrapers/leumi-card.js

+    description: rawTransaction.merchantName.trim(),
+    memo: rawTransaction.comments,
+    installments: getInstallmentsInfo(rawTransaction.comments),
+    status: TRANSACTION_STATUS.COMPLETED,


The value of status should be set to TRANSACTION_STATUS.PENDING when transactions are approved yet not charged. See below example:
processedDate is null

Consider checking processedDate and decide on status accordingly.

@esakal
is it a proposal of a new behaviour or an old one that i did not support?
the screenshot you added shows that "paymentDate" is null, but you suggest checking "processedDate" value, please explain

It's a new feature of LeumiCard,I don't think they returned pending transactions in the old API.

Now that they introduce such a use-case we must support so it is not exactly a regression nor a proposal :)

sure :)
can you explain about the the new logic you proposed?
the screenshot you added shows that "paymentDate" is null, but you suggest checking "processedDate" value

How about:

const isPending = rawTransaction.paymentDate === null; return { processedDate: moment(isPending ? rawTransaction.purchaseDate : rawTransaction.paymentDate).toISOString(), status: isPending ? TRANSACTION_STATUS.PENDING : TRANSACTION_STATUS.COMPLETED, };

I don't think that we can leave the processedDate property value null and it make sense to use the same date of the purchase as this transaction in PENDING mode.

Done
Take a look

@esakal are you sure it doesn't make sense to leave it empty? putting the purchaseDate into the processedDate will break the processedDate semantics (as the transaction wasn't actually processed)

@erikash, I tried to keep the same behavior as other scrapers that expose pending transactions (leumi, discount). They seems to set a date also for pending transactions. I think people expect this field to be required and their code might break at runtime once we introduce an optional as null value.

I think the term Pending is a bit misleading, as it doesn't say if its data is final and the transaction pending approval (either will remain or will be removed) or if it means that the transaction data is not final as well and might be modified once approved.

That being said, I don't might setting this value as null but in that case we should mention it in the readme.md file and consider upgrading major version to indicate breaking changes. @eshaham please let us know how you want to handle this one.

@esakal AFAIK pending charges amounts can be changed.. they are used to "hold" funds in cases that the final amount is not yet known (like an open tab in a hotel) or when buying gas in some gas stations... but i'm not sure about the semantics in Leumi Card :(

I liked your idea!
Maybe we can do both:

Keep the contract and publish a minor

In the next major change the default behavior.

@eshaham please advise...

Keep the contract and publish a minor

In the next major change the default behavior.

I agree, let's not make a big change over this PR - I assume people are waiting for this scraper to be fixed ASAP, so let's get it out.

…ordingly

eshaham

@liranso @esakal I think we're missing something important here.
As I mentioned in #217 , we should make REST api calls to get the data and not navigate to the page.
So it should look more like this:

const url = getTransactionsUrl(monthMoment);
const result = await fetchGetWithinPage(page, txnsUrl);

fetchGetWithinPage is a function already included under /helpers/fetch.
This approach would make the scraper work 10 times faster, and will not require any page navigation after the login process.

esakal · 2019-07-13T12:27:09Z

@eshaham indeed no need to fetch the rest content by page navigation.
@liranso can you adjust the code per @eshaham request?
Note that you should use date filter so the following methods can be unified fetchTransactionsForMonth, fetchTransactions and getCurrentTransactions

Let me know if you need my assistance.
Eran.

liranay · 2019-07-13T16:40:05Z

@esakal when i'm using "fetchGetWithinPage" i'm getting "Evaluation failed: [object Object]"
for some reason i'm not able to understand the cause of the error
i'm debugging with webstorm and defined the following node parameters - "-r babel-register"
i'm guessing it's somehow related to babel

** tester.js file in project root folder **
const {createScraper} = require('./src');

const date = new Date();
date.setMonth(date.getMonth() - 1);

(async function() {
    const scraper = createScraper({
        companyId: 'leumiCard',
        startDate: date, 
        combineInstallments: false,
        showBrowser: true,
        verbose: true,
    });
const result = await scraper.scrape({
            username: 'XYZ',
            password: 'XYZ'
        }
    )
})();

do you have an idea?

esakal · 2019-07-14T07:42:37Z

@lielran Hi

esakal when i'm using "fetchGetWithinPage" i'm getting "Evaluation failed: [object Object]"
for some reason i'm not able to understand the cause of the error
i'm debugging with webstorm and defined the following node parameters - "-r babel-register"
i'm guessing it's somehow related to babel

I created a workable scraper. just copy&paste from this gist into your PR and see if it works for you. If it works just update your PR accordingly.

I didn't use the date filters as it seems to break the installments expected results. I created a gist with that version but ended up reverting and using the monthly scrapping concept to get the expected installments behavior.

We will then wait to @eshaham to decide about @erikash question (here) so we will be able to merge it to master soon.

Changes

[x] use fetchGetWithinPage to fetch transactions
[x] in fetchTransactions - filter months to include next month (to fetch all transactions known till today even if part of next month payment).
[x] removed unused methods

liranay · 2019-07-14T19:42:37Z

@esakal
Changed the PR
regarding this line
const allMonths = getAllMonthMoments(startMoment, true);
why do we need the next month?

esakal · 2019-07-15T14:34:09Z

@esakal
Changed the PR
regarding this line
const allMonths = getAllMonthMoments(startMoment, true);
why do we need the next month?

Great, I will test it and hopefully merge it today. Did it work for you with getAllMonthMoments(startMoment, true); set to true?

To answer your question. This library scrape all transactions created till today (provided start time from user).If you set this argument to false it will not scrape next payment transactions. Let say your payment date is the 5th of each month, if you set it to false you will not get any transaction between 6/07 till today. You can easily check this out by setting this value to false if your card payment date is less then the 15th of each month (as today is 15/07). Hope this clarify the reason.

esakal · 2019-07-15T16:07:08Z

@liranso it is such a small world - it was nice seeing you :)
I confirmed that it works and will merge it in the following minutes. Thanks a lot for your contribution!

brafdlog · 2019-07-15T16:40:14Z

Thanks!

esakal · 2019-07-15T16:42:46Z

@eshaham I merged the scraper into master, please deploy a new version we you can. Thanks!

liranay · 2019-07-16T07:00:46Z

@esakal :)

brafdlog · 2019-07-17T06:00:40Z

@eshaham can you publish a new version to npm?

eshaham · 2019-07-17T10:13:58Z

@brafdlog @esakal @liranso done 👍
thanks for this great PR @liranso

fixed broken leumi card adapter - using REST calls for each month to …

7e3fce3

…fetch transactions

esakal requested changes Jul 12, 2019

View reviewed changes

taking pending transactions into consideration - adjusting status acc…

9d056a7

…ordingly

eshaham requested changes Jul 13, 2019

View reviewed changes

using fetchGetWithinPage for REST calls

d422ffa

esakal merged commit 08e5cad into eshaham:master Jul 15, 2019

eshaham mentioned this pull request Jul 17, 2019

Leumi card scraper has stopped working #217

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixed broken leumi card adapter #219

fixed broken leumi card adapter #219

liranay commented Jul 12, 2019

esakal left a comment

esakal Jul 12, 2019

liranay Jul 12, 2019 •

edited

Loading

esakal Jul 12, 2019

liranay Jul 12, 2019

esakal Jul 12, 2019

liranay Jul 13, 2019

erikash Jul 13, 2019

esakal Jul 14, 2019

erikash Jul 14, 2019

eshaham Jul 15, 2019

eshaham left a comment

esakal commented Jul 13, 2019 •

edited

Loading

liranay commented Jul 13, 2019 •

edited

Loading

esakal commented Jul 14, 2019 •

edited

Loading

liranay commented Jul 14, 2019

esakal commented Jul 15, 2019

esakal commented Jul 15, 2019

brafdlog commented Jul 15, 2019

esakal commented Jul 15, 2019 •

edited

Loading

liranay commented Jul 16, 2019

brafdlog commented Jul 17, 2019

eshaham commented Jul 17, 2019

fixed broken leumi card adapter #219

fixed broken leumi card adapter #219

Conversation

liranay commented Jul 12, 2019

esakal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liranay Jul 12, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eshaham left a comment

Choose a reason for hiding this comment

esakal commented Jul 13, 2019 • edited Loading

liranay commented Jul 13, 2019 • edited Loading

esakal commented Jul 14, 2019 • edited Loading

Changes

liranay commented Jul 14, 2019

esakal commented Jul 15, 2019

esakal commented Jul 15, 2019

brafdlog commented Jul 15, 2019

esakal commented Jul 15, 2019 • edited Loading

liranay commented Jul 16, 2019

brafdlog commented Jul 17, 2019

eshaham commented Jul 17, 2019

liranay Jul 12, 2019 •

edited

Loading

esakal commented Jul 13, 2019 •

edited

Loading

liranay commented Jul 13, 2019 •

edited

Loading

esakal commented Jul 14, 2019 •

edited

Loading

esakal commented Jul 15, 2019 •

edited

Loading