Scopes with foreign key #9735

u9r52sld · 2018-07-29T03:56:53Z

Pull Request check-list

Please make sure to review and check all of these items:

Does npm run test or npm run test-DIALECT pass with this change (including linting)?
Does the description below contain a link to an existing issue (Closes #[issue]) or a description of the issue you are solving?
Have you added new tests to prevent regressions?
Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
Did you follow the commit message conventions explained in CONTRIBUTING.md?

Description of change

Confirming options after foreign key is added.

See issue: #6245

codecov · 2018-07-29T04:03:13Z

Codecov Report

Merging #9735 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #9735      +/-   ##
==========================================
+ Coverage    96.3%   96.32%   +0.01%     
==========================================
  Files          63       63              
  Lines        9413     9408       -5     
==========================================
- Hits         9065     9062       -3     
+ Misses        348      346       -2

Impacted Files	Coverage Δ
lib/model.js	`96.8% <100%> (+0.11%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 122bd17...df69b87. Read the comment docs.

sushantdhiman · 2018-07-30T15:10:19Z

Thanks for this fix. I want to do some tests on my local setup after that I will merge this PR

sushantdhiman · 2018-08-01T06:39:04Z

Tested this locally, this wont work with defaultScope but I have no good suggestion to handle that case right now. Apart from that it works well with other scopes defined by options.scope

@u9r52sld I can merge this in current state or would you like to fix defaultScope case as well?

u9r52sld · 2018-08-01T08:44:00Z

@sushantdhiman Thank you for testing. Of course I'll fix defaultScope case. Please hold on.

u9r52sld · 2018-08-07T02:52:36Z

Someone please tell me why the codecov/project failed. What should we do?

sushantdhiman · 2018-08-07T05:28:25Z

Someone please tell me why the codecov/project failed. What should we do?

You can ignore it in this case.

LGTM, I'll again test a few things locally and merge

sushantdhiman · 2018-08-15T04:06:49Z

Finally got the chance to test this (basically was not active with Sequelize for past week). Looks like attributes mixing is broken after this, consider this case

const Model = sequelize.define('model', {
  name: Sequelize.STRING,
  value: Sequelize.INTEGER,
  password: Sequelize.STRING
}, {
    defaultScope: {
      attributes: {
        exclude: ['password']
      }
    },
    scopes: {
      aScope: {
        attributes: {
          exclude: ['value']
        }
      }
    }
  });

console.log(
  Model._scope
);

console.log(
  Model.scope('aScope', 'defaultScope')._scope
);

{ attributes: { exclude: [ 'password' ] } } // correct
{ attributes: { exclude: [ 'value' ] } } // incorrect, exclude should include both `password`, `value`

sushantdhiman · 2018-08-21T16:10:57Z

I think we should conform scope just before merging them that will solve all concerns I have raised so far. Just conform scopes before merging with other scopes and keep merge/assign logic same as before (as much as possible), otherwise it can become an unintended breaking change.

u9r52sld · 2018-08-22T01:03:53Z

I think we should conform scope just before merging them that will solve all concerns I have raised so far

Could you please tell me the reason?

I think we should not conform scopes like exclude or include before queries are executed because it can be added new foreign_key fields by associations after that.

sushantdhiman · 2018-08-22T06:13:32Z

May be we can merge attributes in that style, but not for include.

Conforming will convert both include: [{ model: ABC }] and include: [ABC] to same format so we don't have duplicate associations.

Conform standardise includes and other properties of scope so merge is predictable

sushantdhiman · 2018-08-31T09:29:40Z

@u9r52sld Thanks, now it looks almost ready.

Some concerns related to merge strategy changes, we have discussed it in #9087 for a few months. I have asked participants from that discussion to share their ideas here. I don't use scopes in my projects so not sure what merge strategy works well. I hope we can figure out something that works for everyone.

P.S. I'll be out for a few days, after that we can complete our discussion and merge this to v5

papb · 2018-08-31T12:14:42Z

Hello, I'm the one who opened #9087, I believe sushantdhiman wanted me to take a look here because this is also related to scopes. I will try to take a look as soon as possible (in less than 48 hours probably). In the meantime, if you could read my concerns on #9087 as well that would be great :) I'll be happy to discuss further and answer questions.

papb · 2018-09-01T13:42:17Z

Hello again, I took a closer look, this PR seems tangentially related to my issue #9087 because it is also related to scopes, but if I'm not mistaken that's about it. @u9r52sld, what do you think? If you could take a look as well that would be great.

sushantdhiman · 2018-09-04T06:36:00Z

@u9r52sld @papb wrt to changes in this PR we should only target how scopes are merged among themselves. That is under scope of this PR.

For overall scope overhaul next we need to decide how scopes are merged with finder options, but that should be a different PR.

This PR is changing both scope's internal representation (like attributes are no longer converted from include/exclude to plain format etc) and strategy how to merge with other scopes. I want your suggestion (possibly others may be), if this current merge strategy (for merging scopes among themselves) is ok, or need any more changes (which btw looks good to me)

papb · 2018-09-04T21:40:12Z

Oh, the way that scopes are merged among themselves does concern my issue as well.

Currently, two scopes with different includes do not merge correctly. I show this in my issue #9087.

Once I have more time, I can prepare an specific test that fails. But I believe this should be straightforward from my issue.

If anyone has questions let me know. In the weekend I'll have time to write the failing test.

papb · 2018-09-10T00:47:05Z

Hello, I didn't have time to write the test as I was thinking, but please take a look at this other test that I wrote. It is about a merge of a scope with the findAll options, but the same idea for merging two scopes fails the exact same way.

Let me know if you have any other questions or want clarifications.

u9r52sld · 2018-09-14T05:57:48Z

@sushantdhiman @papb I'm sorry for the late reply. Thank you for your concern!
I'll check issue #9087 and fix it.

papb · 2018-09-14T22:04:58Z

@u9r52sld Thank you very much! Let me know if you need any further clarifications.

sushantdhiman · 2018-09-30T03:47:11Z

test/unit/model/scope.test.js

      }, { override: true });

      expect(Company._scope).to.deep.equal({
        include: [{ model: Project }]
      });
    });

-    it('works with exclude and include attributes', () => {
+    it('should not work with exclude and include attributes', () => {


should keep exclude and include attributes

sushantdhiman · 2018-09-30T03:54:14Z

lib/model.js

-
-      // Reverse so we consider the latest include first.
-      // This is used if several scopes specify the same include - the last scope should take precedence
-      for (const scopeInclude of scope.include.reverse()) {


What is the new merge strategy now (wrt https://github.com/sequelize/sequelize/blob/master/docs/scopes.md#merging), as far as I can tell it is still the same?

Yeah, it's almost the same. I just updated for issues #9087,#6245

For example, current merge strategy is as follows.

Company.addScope('scopeA', { where: { name: 'A', something: true }, attributes: { include: ['name'] }, include: [ { model: Project, where: { name: 'A', something: true }, }, Project ], limit: 5, order: [ ['name', 'DESC'], ] }) Company.addScope('scopeB', { where: { name: 'B' }, attributes: { exclude: ['something'] }, include: [ { model: Project, as: 'Pj' }, { model: Project, where: { name: 'B' } }, ], limit: 10, order: [ ['something', 'ASC'], ] }) expect(Company.scope(['scopeA', 'scopeB'])._scope).to.deep.equal({ where: { name: 'B', something: true }, // 'where' is shallowly merged attributes: { // just override exclude: ['something'] }, include: [ // 'include' is shallowly merged with the same model and alias(as) { model: Project, where: { name: 'B', something: true }, }, { model: Project, as: 'Pj' }, ], limit: 10, // just override order: [ // Array is united ['name', 'DESC'], ['something', 'ASC'] ] })

Hmm I thought we discussed to have attributes.include / exclude merge #9735 (comment)

Oh, i'm sorry. I fixed it. e31be36

Company.addScope('scopeA', { attributes: { include: ['name'] } }) Company.addScope('scopeB', { attributes: { include: ['something'], exclude: ['password'] } }) expect(Company.scope(['scopeA', 'scopeB'])._scope).to.deep.equal({ attributes: { // merged include: ['name', 'something'], exclude: ['password'] } })

…options

…clude

an unnecessary file

papb · 2018-10-27T20:38:01Z

This is looking very good 👍

Thanks @u9r52sld! I tested your changes against my examples in #9087 and I'm very happy with the results. I also tested more complicated and nested combinations and they also worked as I'd like. Thank you very much! 😁

sushantdhiman · 2018-10-28T11:19:09Z

Thanks @u9r52sld for delivering after this 3 months long review process :) & thanks @papb for your additional comments and review

harpcio · 2018-11-24T09:07:58Z

Are there any chances for this bugfix to be in v4?

papb · 2018-11-24T13:20:26Z

@harpcio I'd guess that probably not, because although I agree that the previous behavior was wrong and that this can be seen as a "bug fix", in fact this is a breaking change, and breaking changes should not be added without a change in the major version number (following semver).

papb · 2019-01-17T16:14:49Z

Extra documentation for this PR was added in #10312

sushantdhiman mentioned this pull request Jul 30, 2018

defaultScope breaks association fetching #9736

Closed

sushantdhiman added the pr:needs work label Aug 15, 2018

sushantdhiman mentioned this pull request Aug 31, 2018

Scopes' includes are not shallowly merged as the documentation claims #9087

Closed

sushantdhiman reviewed Sep 30, 2018

View reviewed changes

sushantdhiman added status: in discussion For issues and PRs. Community and maintainers are discussing the applicability of the issue. and removed pr:needs work labels Oct 15, 2018

u9r52sld added 6 commits October 25, 2018 14:20

fix(scopes-with-foreign-key): returning foreign key with scopes

b5e135c

fix(scopes-with-foreign-key): with defaultScope

692b58d

fix(scopes-with-foreign-key): modify test

404f2a9

fix(scopes-with-foreign-key): scope method should not expand attributes

19361ec

fix(scopes-with-foreign-key): addScope function either

95adff6

fix(scopes-with-foreign-key): unite attributes with array

82cc793

u9r52sld added 11 commits October 25, 2018 14:20

fix(scopes-with-foreign-key): handle symbolic Sequezlie.OP

2ce55c4

fix(scopes-with-foreign-key): conform include before merging

c50527d

fix(scopes-with-foreign-key): enable to merge includes with find options

7267514

fix(scopes-with-foreign-key): minor adjustment

b90f445

fix(scopes-with-foreign-key): fix lint

bafba3e

fix(scopes-with-foreign-key): change merge logic for scopes and find …

78848cb

…options

fix(scopes-with-foreign-key): refactor, fix lint

c4e324c

fix(scopes-with-foreign-key): add test

c7e26ac

fix(scopes-with-foreign-key): enable to merge attributes.include / ex…

e31be36

…clude

fix(scopes-with-foreign-key): remove

6b2f84a

an unnecessary file

fix(scopes-with-foreign-key): remove unnecessary codes

df69b87

sushantdhiman approved these changes Oct 28, 2018

View reviewed changes

sushantdhiman merged commit 88a340d into sequelize:master Oct 28, 2018

sushantdhiman mentioned this pull request Dec 12, 2018

Scoped associations overwrite scoped attributes in model, regression from version 3 #10254

Closed

papb mentioned this pull request Jan 3, 2019

Docs and more tests for #9087 #10312

Merged

5 tasks

tsasaki609 mentioned this pull request Jan 21, 2019

[5.0.0-beta.14][Regression] mixed where clauses in include #10123

Closed

SimonSchick mentioned this pull request Mar 22, 2019

TypeError: include.attributes.map is not a function #10399

Closed

papb mentioned this pull request Apr 2, 2019

Bug: Regression: problem merging simple includes from scopes #10688

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scopes with foreign key #9735

Scopes with foreign key #9735

u9r52sld commented Jul 29, 2018 •

edited

codecov bot commented Jul 29, 2018 •

edited

sushantdhiman commented Jul 30, 2018

sushantdhiman commented Aug 1, 2018

u9r52sld commented Aug 1, 2018

u9r52sld commented Aug 7, 2018

sushantdhiman commented Aug 7, 2018

sushantdhiman commented Aug 15, 2018

sushantdhiman commented Aug 21, 2018

u9r52sld commented Aug 22, 2018

sushantdhiman commented Aug 22, 2018

sushantdhiman commented Aug 31, 2018

papb commented Aug 31, 2018

papb commented Sep 1, 2018

sushantdhiman commented Sep 4, 2018

papb commented Sep 4, 2018

papb commented Sep 10, 2018

u9r52sld commented Sep 14, 2018

papb commented Sep 14, 2018

sushantdhiman Sep 30, 2018

sushantdhiman Sep 30, 2018

u9r52sld Oct 25, 2018 •

edited

sushantdhiman Oct 25, 2018

u9r52sld Oct 25, 2018

papb commented Oct 27, 2018

sushantdhiman commented Oct 28, 2018 •

edited

harpcio commented Nov 24, 2018

papb commented Nov 24, 2018

papb commented Jan 17, 2019

Scopes with foreign key #9735

Scopes with foreign key #9735

Conversation

u9r52sld commented Jul 29, 2018 • edited

Pull Request check-list

Description of change

codecov bot commented Jul 29, 2018 • edited

Codecov Report

sushantdhiman commented Jul 30, 2018

sushantdhiman commented Aug 1, 2018

u9r52sld commented Aug 1, 2018

u9r52sld commented Aug 7, 2018

sushantdhiman commented Aug 7, 2018

sushantdhiman commented Aug 15, 2018

sushantdhiman commented Aug 21, 2018

u9r52sld commented Aug 22, 2018

sushantdhiman commented Aug 22, 2018

sushantdhiman commented Aug 31, 2018

papb commented Aug 31, 2018

papb commented Sep 1, 2018

sushantdhiman commented Sep 4, 2018

papb commented Sep 4, 2018

papb commented Sep 10, 2018

u9r52sld commented Sep 14, 2018

papb commented Sep 14, 2018

sushantdhiman Sep 30, 2018

Choose a reason for hiding this comment

sushantdhiman Sep 30, 2018

Choose a reason for hiding this comment

u9r52sld Oct 25, 2018 • edited

Choose a reason for hiding this comment

sushantdhiman Oct 25, 2018

Choose a reason for hiding this comment

u9r52sld Oct 25, 2018

Choose a reason for hiding this comment

papb commented Oct 27, 2018

sushantdhiman commented Oct 28, 2018 • edited

harpcio commented Nov 24, 2018

papb commented Nov 24, 2018

papb commented Jan 17, 2019

u9r52sld commented Jul 29, 2018 •

edited

codecov bot commented Jul 29, 2018 •

edited

u9r52sld Oct 25, 2018 •

edited

sushantdhiman commented Oct 28, 2018 •

edited