Run lingui extract for specified files

[dephiros]

Is your feature request related to a problem? Please describe.
I am trying to run lingui extract in CI steps to warn developer if they forget to run lingui extract. However running lingui extract over the entire project takes ~10mins

Describe the solution you'd like
lingui extract should take an optional glob that narrow down the list of files the command run on

Describe alternatives you've considered
lingui v3 seems to include a watch option. Maybe this is related?

Additional context

[tricoder42]

Have you considered srcPathDirs and srcPathIgnorePatterns?

[dephiros]

@tricoder42 , I will try to explain the use case a bit better. Let's say my project have these files:
src/a.js
src/b.js
src/c.js
src/d.js

After running lingui extract the first time with srcPathDirs, we will have a messages.json with messages from all the files. However that run get expensive as we add more files(~10mins)

What we want to do is to create a wrapper command that:

• developer changes src/a.js
• Run lingui extract for a.js (this should be a lot faster)
  -messages.json should contains all the existing translation + new message from a.js

This would be helpful for incremental run or CI check run. I think this would also relate to the work that requires to implement watch mode?

From my understanding, srcPathDirs and srcPathIgnorePatterns would not help in this case? even if we change the config before each run?

[tricoder42]

Make sense, thank you for explaining!

srcPathDirs and srcPathIgnorePatterns might help to speed up the extraction, but this is completely different issue.

The tricky part here will be checking if message is obsolete. Since we don't extract messages from all files, we need to check origin of messages: if there're any messages in previous catalog extracted from specified files, but now they're missing, then they're obsolete. It'll require some refactoring in extract script. I would wait until #326 is merged and then implement it on next branch, otherwise we would have to deal with lot of merge conflicts :/

Are you insterested in working on PR?

[dephiros]

Hi @tricoder42, thank you for reopening the issue.
I will try to work on this but a bit busy at the moment so it might be slow going

[tricoder42]

No worries. I already merged #326, so you can start it from next branch.

It's gonna take a while to figure out how it should work, especially with new catalogs API.

With new catalogs, you might have messages split between several files:

src/
  componentA/
    locales/en/messages.po
    ComponentA.js
  componentB/
    locales/en/messages.po
    ComponentB.tjs

It means, when you run lingui extract src/componentB/, it should only updates messages inside src/componentnB/locales/*/messages.po catalogs.

When you run lingui extract src/, it should extract & updated messages in both catalogs.

There's already getCatalogForFile function (see example usage in @lingui/loader), but I believe more efficient would be:

1. Get a list of all catalogs - getCatalogs
2. Instead of collecting messages from all source files, collect from specified files only which belongs to each catalog.
3. Figure out how to partially merge with existing catalogs, probably comparing message.origin.

[dephiros]

I have good news. I just talked to people at my company and there is interest to support this which means things might be able to move a bit faster.

@tricoder42 , do you have an idea for v3 release timeline? If it is still a bit a way, we might also be interested in backporting this to v2 so we can start using it soon in our company

[tricoder42]

@dephiros That would be great!

I set the deadline for milestone to beginning of June. I believe the beta could be ready in 1-2 weeks and then I want to test it on several projects (CRA, Next, Gatsby) to find obvious bugs.

Backporting is always an option, just keep in mind that catalogs are a bit different. In v2, you always have just one catalog, while in v3 you can have many. That might complicate things a bit. On the other hand, lot of things will be reused, like matching obsolete messages based on origin.

[tesseralis]

@tricoder42 @dephiros Is this feature still being worked on? I'm running into the same issue as well on my site and this would really help out!

[dephiros]

Sorry I haven't been available to work on this ☹️

…

On Wed, Feb 26, 2020, 14:10 Nat Alison ***@***.***> wrote: @tricoder42 <https://github.com/tricoder42> @dephiros <https://github.com/dephiros> Is this feature still being worked on? I'm running into the same issue as well on my site and this would really help out! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#510?email_source=notifications&email_token=AARPAWJF24UXMN6ZXEBRNRTRE3SGXA5CNFSM4HOXRXMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENCCZBQ#issuecomment-591670406>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARPAWKZI2ALAY5WRUAM7TTRE3SGXANCNFSM4HOXRXMA> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[levinqdl]

Hi, @tricoder42. Is this available in latest version?

[tricoder42]

@levinqdl It isn't, unfortunatelly. The issue was closed due to lack of activity. Feel free to send a PR if you want to bring it to live. There're few problems to solve which I've desribed in this comment #510 (comment)

[nuragic]

Hi! I was looking for this feature as well, we're experiencing higher extract times right now (more code, more messages)...

FYI we use lingui CLI within our development workflow using lint-staged... we run (commit hook) eslint for linting and after that we run lingui extract and it works great; the only difference is that eslint do accept a list of files (so it actually lint only the staged files) but lingui does not (yet)😄 ... in this case, apart from the performance issue, there's an additional problem, you see: we are including messages in one commit that are not part of that commit...

[semoal]

I could work on it this week at night, lets go

[semoal]

I think I fixed this issue with two lines of code .. I'll create a pull request once correctly tested

Some performance testing:

New version with files param:

yarn extract src 1.47s user 0.26s system 111% cpu 1.709 total

Actual code branch without files filter:

yarn extract 4.19s user 0.33s system 135% cpu 3.345 total

[nuragic]

Wow! You rock! 🤘 Thanks! Looking forward to update

[semoal]

😱 Released 3.3.0 with this fix/feature introduced!

npx update-by-scope @lingui