Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LPS-86129 Removes temporary SF supression #68246

Closed
wants to merge 8 commits into from

Conversation

jbalsas
Copy link

@jbalsas jbalsas commented Feb 19, 2019

No description provided.

@liferay-continuous-integration
Copy link
Collaborator

CI is automatically triggering "ci:test:sf" and "ci:test:relevant" for this pull to run Source Formatter and relevant tests.

Comment "ci:test" to run the full PR Tester for this pull.

@liferay-continuous-integration
Copy link
Collaborator

✔️ ci:test:sf - 1 out of 1 jobs passed in 2 minutes 42 seconds 734 ms

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: 0f1ff4d66f9a654ba4446fb0cbba8d87248d791f

Sender Branch:

Branch Name: pr-1658
Branch GIT ID: 1bb1ccc65d71ee7586f9fa6b5604ef79a1d35dc5

1 out of 1jobs PASSED
1 Successful Jobs:
For more details click here.

@liferay-continuous-integration
Copy link
Collaborator

❌ ci:test:relevant - 41 out of 87 jobs passed in 1 hour 54 minutes 48 seconds 598 ms

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: 0f1ff4d66f9a654ba4446fb0cbba8d87248d791f

Copied in Private Modules Branch:

Branch Name: master-private
Branch GIT ID: da6217fa3ca8202155a9b41d981cac49efc62fa9

41 out of 87 jobs PASSED

46 Failed Jobs:

41 Successful Jobs:
For more details click here.

Failures unique to this pull:

  1. test-portal-acceptance-pullrequest-batch(master)/functional-smoke-tomcat90-mysql57-jdk11_open/0
    Job Results:

    0 Tests Passed.
    1 Test Failed.

    1. AXIS_VARIABLE=0,label_exp=!master #280553
          [junit] 	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
          [junit] 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
          [junit] 	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
          [junit] 	at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
          [junit] 	at java.nio.file.Files.newDirectoryStream(Files.java:525)
          [junit] 	at com.liferay.portal.log.assertor.PortalLogAssertorTest.testScanOSGiLog(PortalLogAssertorTest.java:62)
          [junit] 
          [junit] 
          [junit] Test com.liferay.portal.log.assertor.PortalLogAssertorTest FAILED
           [echo] The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test-batch.xml:340: The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test-batch.xml:1234: /opt/dev/projects/github/liferay-portal/build-test-batch.xml:1238: The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test-tomcat.xml:46: The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test-tomcat.xml:54: The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test.xml:10889: The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test.xml:3843: The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test.xml:3962: 
           [echo] exec returned: 1
           [echo] The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test.xml:4589: The following error occurred while executing this line:
           [echo] /opt/dev/projects/github/liferay-portal/build-test.xml:4730: Unable to find startup message after 30 minutes.
           [echo] ${jstack.output}
            [get] Getting: http://test-1-14/job/test-portal-acceptance-pullrequest-batch(master)/AXIS_VARIABLE=0,label_exp=!master/280553//consoleText
            [get] To: /opt/dev/projects/github/liferay-portal/20190219180542868.txt
         [delete] Deleting: /opt/dev/projects/github/liferay-portal/20190219180542868.txt
         [delete] Deleting: /opt/dev/projects/github/liferay-portal/null826114183.properties
  2. test-portal-acceptance-pullrequest-batch(master)/pmd-jdk8
    Job Results:

    0 Tests Passed.
    1 Test Failed.

    1. AXIS_VARIABLE=0,label_exp=!master #294495
      [beanshell] local.git.branch.cached=true
      [beanshell] github.origin.name=liferay
      [beanshell] 
         [delete] Deleting: /opt/dev/projects/github/liferay-jenkins-ee/json.data
           [echo] curl --output /opt/dev/projects/github/liferay-jenkins-ee/json.data --silent --write-out %{http_code} 'http://cloud-10-0-0-31/osb-jenkins-web/map/test-1-5/test-portal-acceptance-pullrequest%28master%29/3818/test-portal-acceptance-pullrequest%28master%29/git.portal.properties'
           [exec] 200
         [delete] Deleting: /opt/dev/projects/github/liferay-jenkins-ee/commands/20190219174700030
      [beanshell] 
      [beanshell] ##
      [beanshell] ## http://cloud-10-0-0-31/osb-jenkins-web/map/test-1-5/test-portal-acceptance-pullrequest%28master%29/3818/test-portal-acceptance-pullrequest%28master%29/git.portal.properties
      [beanshell] ##
      [beanshell] 
      [beanshell] github.upstream.branch.name=master
      [beanshell] github.sender.branch.sha=1bb1ccc65d71ee7586f9fa6b5604ef79a1d35dc5
      [beanshell] github.receiver.username=brianchandotcom
      [beanshell] github.sender.branch.name=pr-1658
      [beanshell] github.pull.request.number=68246
      [beanshell] github.upstream.branch.sha=0f1ff4d66f9a654ba4446fb0cbba8d87248d791f
      [beanshell] github.sender.username=jbalsas
      [beanshell] local.git.branch.cached=true
      [beanshell] github.origin.name=jbalsas
      [beanshell] 
         [delete] Deleting: /opt/dev/projects/github/liferay-jenkins-ee/json.data
           [echo] WARNING: pmd-jdk8 does not match existing test types (blade-samples,database-upgrade-client,functional,jsf,plugins,portal-startup,service-builder,tck,modules-integration,integration,modules-unit,unit)
           [echo] WARNING: pmd-jdk8 does not match existing test types (blade-samples,database-upgrade-client,functional,jsf,plugins,portal-startup,service-builder,tck,modules-integration,integration,modules-unit,unit)
      

      BUILD FAILED
      /opt/dev/projects/github/liferay-jenkins-ee/commands/build-common.xml:10412: The following error occurred while executing this line:
      /opt/dev/projects/github/liferay-portal/build-test-batch.xml:5275: 1 PMD violation(s) were found.

      PMD VIOLATION: /opt/dev/projects/github/liferay-portal/modules/private/apps/portal-workflow/portal-workflow-metrics-rest-impl/src/main/java/com/liferay/portal/workflow/metrics/rest/internal/graphql/mutation/v1_0/Mutation.java:30 > Avoid unused local variables such as 'bundle'. (PMD Rule: UnusedLocalVariable)
      For more information, please see https://pmd.github.io/pmd-6.8.0/pmd_rules_java_bestpractices.html#unusedlocalvariable.


Failures in common with acceptance upstream results at e5f49f1:
  1. ...

@brianchandotcom
Copy link
Owner

Merged. Thank you.
View total diff: 9b1d062...ab725c2

@brianchandotcom
Copy link
Owner

Merged with minor changes in upstream @jbalsas @izaera

@Preston-Crary can you take a look at

modules/apps/frontend-js/frontend-js-loader-modules-extender-api/src/main/java/com/liferay/frontend/js/loader/modules/extender/npm/NPMResolverUtil.java

you're either going to think it's genius.. or it will make you cringe. But I pushed them to do something like it because they were writing the same class over and over again.

Thx

@izaera
Copy link

izaera commented Feb 20, 2019

you're either going to think it's genius.. or it will make you cringe. But I pushed them to do something like it because they were writing the same class over and over again.

🤔

I'm not sure what looks strange in that class. The only problem with NPMResolvers is that they are not tight to OSGi lifecyle but to web context lifecycle: they should not appear until a web context and its related NPM objects have been instantiated. So, you have to wait for all of them to be available, and that's what the NPMResolvedPackageNameRegistrar does.

That said, the new NPMResolverUtil could be made to work without caring about the webcontext lifecycle because you can obtain everything from the OSGi bundle. But, given that we already have a tracker for the other model (webcontext+bundle+npm object) I thought reusing it would be easier and safer than making another isolated setup.

Another possibility is attaching the NPMResolver to the web context (in addition to the npmResolvedPackageName variable) but given that the use of NPMResolverUtil will almost always happen in the OSGi domain (because you use it to resolve modules from your OSGi bundle) I thought it would be better to write it like that. We can also do both if you feel it's needed.

In the end, what happens is that you have two scenarios where you need to use the NPMResolver:

  1. From any OSGi bundle (for example infrastructure ones) to obtain npm modules living in the bundle of the request's web context (the top level user bundle).
  2. From any OSGi bundle to obtain npm modules contained in those bundles (would be like accessing your own resources).

Case 1 is typical in scenarios where infrastructure code wants to pull npm modules from the top level user bundle to use them in some place it is rendering.

Case 2 is typical in scenarios where an infrastructure bundle contains an npm module that it wants to use to provide some feature to top level user code.

However, if you think we can refine it I'm open to feedback. I can think of more ways to fix it but I'm not sure of the final goals (I know we want to simplify it, but I'm unsure about how) so it's difficult for me to decide between the alternatives... :-)

@Preston-Crary
Copy link

@izaera NPMResolvedPackageNameRegistrar seems backwards to me. It's something the bundle needs but isn't available until the bundle is active, it should be the other way around.

The problem is we don't have a good way to handle OSGi services from web and taglib modules since they operate in a different worlds.

@shuyangzhou and I have an idea of something we can do to make referencing dependencies from web modules easier. It'll take some time to be sure it will work and more time to implement but in the end it could fix the resolution problems and require less code in web modules.

@izaera
Copy link

izaera commented Feb 21, 2019

@Preston-Crary It's not really the Bundle what it depends on, it's the processing of the bundle by the NPMRegistry so, even though the bundle needs the NPMResolver it cannot use it until it has been processed by the NPMRegistry. This may look unstable, but in real life it should not be a problem because all NPMResolver uses are done far after the bundle has loaded. But I agree that it can lead to problems in corner cases.

Let me explain precisely what the NPMResolver does to see if we can think of something better.

What we need is to resolve an "npm module name" (which is something like a-fancy-framework/bootstrap) to make the first call to Loader.require. So, every time someone wants to run Javascript, the first require call is kind of an special bootstrap call that "introduces" you in the Javascript world.

Why is that call special? Because a-fancy-framework/bootstrap is not fully defined. If you see the module name, it is made of a package name (a-fancy-framework) and the path to the module JS file. But packages are not defined unless they have a version. So, what we need to get into the Javascript world is something like a-fancy-framework@1.2.0/bootstrap which, this time, is fully defined.

The first approach we did was to specify the version numbers directly. So, when we had (in a JSP for example) something like:

<aui:script require="a-fancy-framework/bootstrap"/> 

we wrote:

<aui:script require="a-fancy-framework@1.2.0/bootstrap"/>

The problem with this was that, whenever you changed the version number of a-fancy-framework you needed to change all you JSP files. Also, 99% of the time a-fancy-framework was, in reality, the name of the very same project that hosted the JSP (which is equal to the OSGi bundle name by our convention). So, it looked very redundant and error prone.

Then, we decided to introduce the NPMResolver to make these module resolutions easier. Because the module name a-fancy-framework/bootstrap may mean different things depending on what bundle you are (this is because bundle A may use, say, version 1.2.0 of a-fancy-framework and bundle B may use a different one, say 2.1.0) NPMResolver instances are not a singleton, but created by an OSGi factory that assigns precisely one NPMResolver per OSGi bundle. This is important, because people tends to think that NPMResolver is a singleton when it is not. And also, again: it cannot be (at least in this universe ;-P).

After that, because NPMResolvers where all over the place we created the concept of npmResolvedPackageName, which is an implicit JSP variable containing the resolved package name of the OSGi bundle hosting the JSP file. Obviously, because you need to automagically infer the OSGi bundle from the JSP (to obtain the NPMResolver; remember you cannot resolve npm module names without an starting bundle) the mechanism for npmResolvedPackageName has to derive the OSGi bundle from the request (the only global thing it has access to). Thus, we made the NPMResolverJSBundleTracker and the NPMResolvedPackageNameRegistrar, which are the objects that wait for:

  1. the WebContext (to attach the npmResolvedPackageName variable to it so that it can be retrieved from the JSP)
  2. the JSBundle (not the OSGi Bundle)

Only when both are ready, we can assign an NPMResolver to a WebContext because before that we cannot access the information needed to make the resolution.

This worked for 99% of the scenarios, but then it came the next problem. Even though 99% of the time you want to refer to a JS module inside the same bundle where your code (JSP or Java) lives, sometimes infrastructure code wants to load JS modules not from the infrastructure bundles, but from the bundles calling the infrastructure. Other times, infrastructure code wants to load JS modules from its own modules. And it is very probable that, in a single class of JSP, the infrastructure code wants to refer to both type of modules (the ones it owns, and the ones owned by the calling bundle).

This could have been done simply by injecting the NPMResolver in the infrastructure bundle. That way, infrastructure code would use its own NPMResolver to resolve its own JS modules and the npmResolvedPackageName to resolve the caller's JS modules. But because we don't want to have NPMResolvers hanging around, I created the NPMResolverUtil which leverages the NPMResolverJSBundleTracker (but not the NPMResolvedPackageNameRegistrar because this time we don't need the WebContext) to get the NPMResolver associated to the calling bundle.

Of course, if you call NPMResolverUtil just after OSGi has made the bundle available, but before NPMRegistry has processed it to create the JSBundle the util won't be able to acces the NPMResolver (as it is not yet created) and the call will fail. However this is unlikely, because the API is designed so that it is called when it is used, not in advance (though I admit that someone may be tempted to cache resolutions in static fields and his setup may break). But we can always add a note in Javadoc explaining the lifecycle and also, a warning is dumped to the console.

Hope this "short & light" explanation makes it clearer to understand.

P.S: there's another possibility we haven't explored, which is making JSBundles true OSGi services so that we can attach their lifecycle to the rest of the other services. I didn't want to do that from the beginning because there are a lot of npm objects and thought it could overload OSGi, but we can give it a try if we see it may help. Also, there are not so much JSBundle objects for OSGi to cope with them. The real explosion comes with JSPackages namespace in first place and then, JSModules which is orders of magnitude bigger than the JSBundle namespace.

To make this last part clearer, what you have is:

OSGi bundle <-1-------1-> JSBundle <-1------n-> JSPackage <-1-------m-> JSModule

Where usually 1000 > n > 10 and 1000 > n > 1 (but it could vary a lot).

@Preston-Crary
Copy link

@izaera thanks for the explanation.

Of course, if you call NPMResolverUtil just after OSGi has made the bundle available, but before NPMRegistry has processed it to create the JSBundle the util won't be able to acces the NPMResolver (as it is not yet created) and the call will fail.

This is the main part we don't like, and it's not unique to NPM. Any statically accessed service used in a web module may have this problem.

Another part we don't like is "pulling" based registration because each time you need something you have to ask for it which takes some CPU time. It's much better if we can find a "pushing" based solution where the client code is given what it needs to execute it's purpose.

This is a good enough solution for short term and I do not want you to resolve this specifically for NPMRegistry, we want to find a general solution for this problem that can be applied to many other services in web modules. I don't want to go into the details of our plan because we're not sure it'll even work yet.

@izaera
Copy link

izaera commented Feb 22, 2019

@Preston-Crary I see :-).

I've also been thinking about this issue a lot of times. Perhaps having something similar to OSGi DS but attached to the WebContexts would make it.

For example, one could declare what OSGi services need to be attached to the WebContext in the BND files, or in Java classes and then the deployer grabs them and injects them as attributes. It could also make sure that the WebContext does not start unless those services are present.

It's not a complete solution, but most of the time people just need singleton services for their JSPs and right now the portal is full of internal *Provider classes that store services in a static field just for that purpose. We are not even using dynamism.

My two cents...

@jbalsas jbalsas deleted the pr-1658 branch February 22, 2019 09:27
@Preston-Crary
Copy link

@izaera that's very similar to our idea, the main difference is we are looking at preventing web bundles from starting unless they have all their requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants