Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[7.4-only] LPS-122191 Removes unnecessary FlashMagicBytes checks #9540

Closed
wants to merge 3 commits into from

Conversation

izaera
Copy link

@izaera izaera commented Oct 26, 2020

This is the continuation of liferay-appsec#143


Based on the analysis done at Analyze security relevance of FlashMagicBytes check):

DXP uses a FlashMagicBytesUtil to check verify if some files are Adobe Flash movies (regardless of their extension). This was added for securtiy reasons to protect from CSRF attack using uploaded flash files

From @topolik:

[...] this would be relevant as long as we support browsers that supports flash. Once all supported browsers discard flash we can remove. Thanks.

All browsers in our possible compatibility matrix for 7.4 will have dropped Flash by the time we release as per their roadmaps:

  • Chrome: Flash Player blocked as "out of date" (Target: All Chrome versions - Jan 2021)
  • Firefox: In January 2021, Firefox 85 will completely remove Flash support. Adobe will stop shipping security updates for Flash at the end of 2020.
  • Safari: Apple just released the latest Safari Technology preview. It comes with many changes, most notably the removal of support for Adobe Flash.

Based on this, I think it's safe to remove the FlashMagicBytes checks.

The solution:

  • FlashMagicBytesUtil uses a PushbackInputStream to read the first bytes of an InputStream and then simply rewinds it and provides means to access it. That means that when removing the usage, we can simply use the original InputStream to keep the same behaviour.
  • To avoid a major kernel breaking change, we're simply deprecating both FlashMagicBytesUtil and FlashMagicBytesUtilTest instead of deleting them.

The FlashMagicBytesUtil checks were added to prevent [CSRF attack using
uploaded flash files](https://issues.liferay.com/browse/LPS-52121). We
will no longer support Adobe Flash in DXP 7.4.

Adobe will stop providing security fixes for the platform in December
2020 and browsers will prevent any content from running past that date.

`FlashMagicBytesUtil` uses a `PushbackInputStream` to read the first bytes
of an `InputStream` and then simply rewinds it and provides means to
access it. That means that when removing the usage, we can simply use
the original `InputStream` to keep the same behaviour.

To avoid a major kernel breaking change, we're simply deprecating both
`FlashMagicBytesUtil` and `FlashMagicBytesUtilTest` instead of deleting
them.
@liferay-continuous-integration
Copy link
Collaborator

To conserve resources, the PR Tester does not automatically run for every pull.

If your code changes were already tested in another pull, reference that pull in this pull so the test results can be analyzed.

If your pull was never tested, comment "ci:test" to run the PR Tester for this pull.

@izaera
Copy link
Author

izaera commented Oct 26, 2020

ci:test:relevant

@izaera
Copy link
Author

izaera commented Oct 26, 2020

ci:test:sf

@liferay-continuous-integration
Copy link
Collaborator

❌ ci:test:sf - 0 out of 1 jobs passed in 39 seconds

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: 912ca54cacc3a3b3ba7ae3b542fd64514d1ff289

Sender Branch:

Branch Name: LPS-122191
Branch GIT ID: 06270a5fcee928fa6465d19f3a9f102b97a469cb

0 out of 1jobs PASSED
For more details click here.
	at org.apache.tools.ant.Task.perform(Task.java:352)
	at org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
	at org.apache.tools.ant.Task.perform(Task.java:352)
	at net.sf.antcontrib.logic.TryCatchTask.execute(TryCatchTask.java:207)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:293)
	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
	at org.apache.tools.ant.Task.perform(Task.java:352)
	at org.apache.tools.ant.Target.execute(Target.java:437)
	at org.apache.tools.ant.Target.performTasks(Target.java:458)
	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1406)
	at org.apache.tools.ant.Project.executeTarget(Project.java:1377)
	at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
	at org.apache.tools.ant.Project.executeTargets(Project.java:1261)
	at org.apache.tools.ant.Main.runBuild(Main.java:857)
	at org.apache.tools.ant.Main.startAnt(Main.java:236)
	at org.apache.tools.ant.launch.Launcher.run(Launcher.java:280)
	at org.apache.tools.ant.launch.Launcher.main(Launcher.java:109)

BUILD FAILED
/opt/dev/projects/github/liferay-jenkins-ee/commands/build-common.xml:12149: The following error occurred while executing this line:
/opt/dev/projects/github/liferay-jenkins-ee/commands/build-test-portal-source-format.xml:18: Sourced file: inline evaluation of: import com.liferay.jenkins.results.parser.AntUtil; import com.lifer . . . '' : TargetError : at Line: 322 : in file: inline evaluation of: import com.liferay.jenkins.results.parser.AntUtil; import com.lifer . . . '' : throw e ;

Target exception: java.lang.RuntimeException: Unable to create local branch izaera-9540-LPS-122191 at 6ca1632c3490059a143e07373850011f63ca77f5
fatal: Not a valid branch point: '6ca1632c3490059a143e07373850011f63ca77f5'.

@izaera
Copy link
Author

izaera commented Oct 26, 2020

ci:test:sf

@liferay-continuous-integration
Copy link
Collaborator

❌ ci:test:relevant - 0 out of 1 jobs passed in 1 minute

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: e069efe223205daa68feca93d7908445b5c8c69a

ci:test:relevant - 0 out of 1 jobs PASSED
For more details click here.

Failures unique to this pull:

  1. test-portal-acceptance-pullrequest(master)
    Job Results:

    0 Jobs Passed.
    1 Job Failed.

    The sender branch SHA could not be found on izaera/LPS-122191. The sender branch may have been force pushed or deleted after the pull request test was initiated.

    fatal: Not a valid branch point: '6ca1632c3490059a143e07373850011f63ca77f5'.

For upstream results, click here.

@liferay-continuous-integration
Copy link
Collaborator

❌ ci:test:sf - 0 out of 1 jobs passed in 0 ms

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: 912ca54cacc3a3b3ba7ae3b542fd64514d1ff289

Sender Branch:

Branch Name: LPS-122191
Branch GIT ID: 06270a5fcee928fa6465d19f3a9f102b97a469cb

0 out of 1jobs PASSED
For more details click here.
     [java] java.lang.Exception: Found 7 formatting issues:
     [java] 1: 'if' is not followed by whitespace., see https://checkstyle.sourceforge.io/config_whitespace.html#WhitespaceAround: ./portal-kernel/src/com/liferay/portal/kernel/servlet/ServletResponseUtil.java 787 (Checkstyle:WhitespaceAroundCheck)
     [java] 2: 'if' is not followed by whitespace., see https://checkstyle.sourceforge.io/config_whitespace.html#WhitespaceAround: ./portal-kernel/src/com/liferay/portal/kernel/servlet/ServletResponseUtil.java 792 (Checkstyle:WhitespaceAroundCheck)
     [java] 3: ')' is followed by whitespace., see https://checkstyle.sourceforge.io/config_whitespace.html#NoWhitespaceAfter: ./portal-kernel/src/com/liferay/portal/kernel/servlet/ServletResponseUtil.java 794 (Checkstyle:NoWhitespaceAfterCheck)
     [java] 4: 'if' is not followed by whitespace., see https://checkstyle.sourceforge.io/config_whitespace.html#WhitespaceAround: ./portal-kernel/src/com/liferay/portal/kernel/servlet/ServletResponseUtil.java 797 (Checkstyle:WhitespaceAroundCheck)
     [java] 5: ')' is followed by whitespace., see https://checkstyle.sourceforge.io/config_whitespace.html#NoWhitespaceAfter: ./portal-kernel/src/com/liferay/portal/kernel/servlet/ServletResponseUtil.java 799 (Checkstyle:NoWhitespaceAfterCheck)
     [java] 6: ')' is followed by whitespace., see https://checkstyle.sourceforge.io/config_whitespace.html#NoWhitespaceAfter: ./portal-kernel/src/com/liferay/portal/kernel/servlet/ServletResponseUtil.java 806 (Checkstyle:NoWhitespaceAfterCheck)
     [java] 7: Truncated message :
     [java] ./portal-kernel/src/com/liferay/portal/kernel/servlet/ServletResponseUtil.java expected:<...nel.util.Validator;
     [java] [
     [java] import java.io.ByteArrayInputStream;
     [java] import java.io.File;
     [java] import java.io.FileInputStream;
     [java] import java.io.IOException;
     [java] import java.io.InputStream;
     [java] import java.io.OutputStream;
     [java] 
     [java] import java.net.SocketException;
     [java] 
     [java] import java.nio.ByteBuffer;
     [java] import java.nio.CharBuffer;
     [java] import java.nio.channels.Channels;
     [java] import java.nio.channels.FileChannel;
     [java] 
     [java] import java.util.ArrayList;
     [java] import java.util.Collections;
     [java] import java.util.List;
     [java] 
     [java] import javax.servlet.ServletOutputStream;
     [java] import javax.servlet.http.HttpServletRequest;

@liferay-continuous-integration
Copy link
Collaborator

Jenkins Build:test-portal-source-format#3537
Jenkins Report:jenkins-report.html
Jenkins Suite: sf
Pull Request:shuyangzhou#9540
Spira Release:/Liferay DXP 7.3/Pull Request/ci:test:sf
Spira Release Build:master - izaera > shuyangzhou - PR#9540 - 2020-10-26[04:23:36]
Spira Jenkins Build:publish-spira-report#6228

@liferay-continuous-integration
Copy link
Collaborator

Jenkins Build:test-portal-source-format#3538
Jenkins Report:jenkins-report.html
Jenkins Suite: sf
Pull Request:shuyangzhou#9540
Spira Release:/Liferay DXP 7.3/Pull Request/ci:test:sf
Spira Release Build:master - izaera > shuyangzhou - PR#9540 - 2020-10-26[04:25:11]
Spira Jenkins Build:publish-spira-report#11904

@izaera
Copy link
Author

izaera commented Oct 26, 2020

ci:test:sf

@izaera
Copy link
Author

izaera commented Oct 26, 2020

ci:test:relevant

@izaera
Copy link
Author

izaera commented Oct 26, 2020

Hey @shuyangzhou . Sorry for the mess with the CI... :-(

I'm sending you this PR so that you can review the last two commits, which affect the backend infrastructure. You can look at the preceding commits at your will, but they are mostly frontend infra stuff.

Given that all the rest has been accepted by the security team, once those two commits are clear, this can be forwarded to Brian for merge.

CCing @jbalsas and @topolik in case they want to add something before this is forwarded to master.

@liferay-continuous-integration
Copy link
Collaborator

✔️ ci:test:stable - 14 out of 14 jobs passed

❌ ci:test:relevant - 44 out of 47 jobs passed in 1 hour 40 minutes

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: e069efe223205daa68feca93d7908445b5c8c69a

Copied in Private Modules Branch:

Branch Name: master-private
Branch GIT ID: 424ab1158c689b11bc409031cffca77c71b5b194

ci:test:stable - 14 out of 14 jobs PASSED
14 Successful Jobs:
ci:test:relevant - 44 out of 47 jobs PASSED
44 Successful Jobs:
For more details click here.

Failures unique to this pull:

  1. test-portal-acceptance-pullrequest-batch(master)/semantic-versioning-jdk8
    Job Results:

    0 Tests Passed.
    1 Test Failed.

    1. AXIS_VARIABLE=0,label_exp=!master #205758

      Please fix semantic versioning on izaera/LPS-122191

           [exec]   PACKAGE_NAME                                       DELTA      CUR_VER    BASE_VER   REC_VER    WARNINGS  
           [exec] = ================================================== ========== ========== ========== ========== ==========
           [exec]   com.liferay.portal.kernel.servlet                  UNCHANGED  8.3.1      8.3.0      8.3.0      EXCESSIVE VERSION INCREASE
           [exec] 	-   version    8.3.0
           [exec] 	+   version    8.3.1
           [exec] [Baseline Report] Mode: diff (persisted)
           [exec] Semantic versioning is incorrect while checking /opt/dev/projects/github/liferay-portal/portal-kernel/portal-kernel.jar against /opt/dev/projects/github/liferay-portal/.gradle/caches/modules-2/files-2.1/com.liferay.portal/com.liferay.portal.kernel/9.12.1/5cf720dd5e5eec99a0b871590d2b6d0079997f9b/com.liferay.portal.kernel-9.12.1.jar

  2. test-portal-acceptance-pullrequest-batch(master)/unit-jdk8
    Job Results:

    2609 Tests Passed.
    1 Test Failed.

    1. AXIS_VARIABLE=0,label_exp=!master #366639
      1. ServletResponseUtilRangeTest.testWriteWithRanges
        junit.framework.AssertionFailedError: array lengths differed, expected.length=10 actual.length=9
        	at com.liferay.portal.kernel.servlet.ServletResponseUtilRangeTest.testWriteWith(ServletResponseUtilRangeTest.java:316)
        	at com.liferay.portal.kernel.servlet.ServletResponseUtilRangeTest.testWriteWithRanges(ServletResponseUtilRangeTest.java:175)
        	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:310)
        	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.executeTest(PowerMockJUnit44RunnerDelegateImpl.java:294)
        	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTestInSuper(PowerMockJUnit47RunnerDelegateImpl.java:127)
        	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:82)
        	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:282)
        	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:207)
        	at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:146)
        	at org...

Failures in common with acceptance upstream results at 37b0a7b:
  1. test-portal-acceptance-pullrequest-batch(master)/unit-jdk8
    Job Results:

    2609 Tests Passed.
    7 Tests Failed.

    1. AXIS_VARIABLE=0,label_exp=!master #366639
      1. LanguagePropertyTest.testValidKeyUserDefaultPortrait
        junit.framework.AssertionFailedError: Invalid values for key "lang.user.default.portrait" are found in: [./portal-impl/src/content/Language_nl_BE.properties, ./portal-impl/src/content/Language_es.properties, ./portal-impl/src/content/Language_fr.properties, ./portal-impl/src/content/Language_ca.properties, ./portal-impl/src/content/Language_nl.properties, ./portal-impl/src/content/Language_pt_PT.properties, ./portal-impl/src/content/Language_de.properties, ./portal-impl/src/content/Language_hu.properties]
        	at com.liferay.portal.kernel.language.LanguagePropertyTest._testValidKey(LanguagePropertyTest.java:286)
        	at com.liferay.portal.kernel.language.LanguagePropertyTest.testValidKeyUserDefaultPortrait(LanguagePropertyTest.java:168)
        
      2. LanguagePropertyTest.testUserNameRequiredFieldNamesSubsetOfUserNameFieldNames
        junit.framework.AssertionFailedError: Required field names are not a subset of user name field names in [./portal-impl/src/content/Language_nl_BE.properties, ./portal-impl/src/content/Language_es.properties, ./portal-impl/src/content/Language_zh_CN.properties, ./portal-impl/src/content/Language_fr.properties, ./portal-impl/src/content/Language_ja.properties, ./portal-impl/src/content/Language_ca.properties, ./portal-impl/src/content/Language_nl.properties, ./portal-impl/src/content/Language_pt_PT.properties, ./portal-impl/src/content/Language_de.properties, ./portal-impl/src/content/Language_hu.properties, ./portal-impl/src/content/Language_it.properties]
        	at com.liferay.portal.kernel.language.LanguagePropertyTest.testUserNameRequiredFieldNamesSubsetOfUserNameFieldNames(LanguagePropertyTest.java:145)
        
      3. LanguagePropertyTest.testValidKeyUserInitialsFieldNames
        junit.framework.AssertionFailedError: Invalid values for key "lang.user.initials.field.names" are found in: [./portal-impl/src/content/Language_nl_BE.properties, ./portal-impl/src/content/Language_es.properties, ./portal-impl/src/content/Language_fr.properties, ./portal-impl/src/content/Language_ca.properties, ./portal-impl/src/content/Language_nl.properties, ./portal-impl/src/content/Language_pt_PT.properties, ./portal-impl/src/content/Language_de.properties]
        	at com.liferay.portal.kernel.language.LanguagePropertyTest._testValidKey(LanguagePropertyTest.java:286)
        	at com.liferay.portal.kernel.language.LanguagePropertyTest.testValidKeyUserInitialsFieldNames(LanguagePropertyTest.java:173)
        
      4. ...

Note that this fixes the problem at
liferay-appsec#143 (comment).

The diagnostic of the problem is explained in the comments of the same
PR of the error, starting at
liferay-appsec#143 (comment).

This commit introduces new methods to distinguish between parameter
types and functionality. The idea is to stop using dynamic type checks
(instanceof) to do one thing or the other and make it explicit in the code
to avoid future errors.

Regarding the two functionalities I mention, there are two ways to
consume the InputStream: one is to skip bytes, the other to directly
seek the position from where to start reading. That's why I created two
different method with two different names.
@izaera
Copy link
Author

izaera commented Oct 26, 2020

ci:test:sf

@izaera
Copy link
Author

izaera commented Oct 26, 2020

ci:test:relevant

@liferay-continuous-integration
Copy link
Collaborator

✔️ ci:test:sf - 1 out of 1 jobs passed in 37 minutes

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: e069efe223205daa68feca93d7908445b5c8c69a

Sender Branch:

Branch Name: LPS-122191
Branch GIT ID: 930b800a2c29c2d149d73936b209673f68e7da1a

1 out of 1jobs PASSED
1 Successful Jobs:
For more details click here.

@liferay-continuous-integration
Copy link
Collaborator

Jenkins Build:test-portal-source-format#4251
Jenkins Report:jenkins-report.html
Jenkins Suite: sf
Pull Request:shuyangzhou#9540
Spira Release:/Liferay DXP 7.3/Pull Request/ci:test:sf
Spira Release Build:master - izaera > shuyangzhou - PR#9540 - 2020-10-26[08:48:30]
Spira Jenkins Build:publish-spira-report#6947

@liferay-continuous-integration
Copy link
Collaborator

✔️ ci:test:stable - 14 out of 14 jobs passed

✔️ ci:test:relevant - 46 out of 47 jobs passed in 1 hour 40 minutes

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: e069efe223205daa68feca93d7908445b5c8c69a

Copied in Private Modules Branch:

Branch Name: master-private
Branch GIT ID: 424ab1158c689b11bc409031cffca77c71b5b194

ci:test:stable - 14 out of 14 jobs PASSED
14 Successful Jobs:
ci:test:relevant - 45 out of 47 jobs PASSED
45 Successful Jobs:
For more details click here.

This pull contains no unique failures.


Failures in common with acceptance upstream results at e069efe:
  1. test-portal-acceptance-pullrequest-batch(master)/unit-jdk8
    Job Results:

    2610 Tests Passed.
    7 Tests Failed.

    1. AXIS_VARIABLE=0,label_exp=!master #339867
      1. LanguagePropertyTest.testValidKeyUserDefaultPortrait
        junit.framework.AssertionFailedError: Invalid values for key "lang.user.default.portrait" are found in: [./portal-impl/src/content/Language_nl_BE.properties, ./portal-impl/src/content/Language_es.properties, ./portal-impl/src/content/Language_fr.properties, ./portal-impl/src/content/Language_ca.properties, ./portal-impl/src/content/Language_nl.properties, ./portal-impl/src/content/Language_pt_PT.properties, ./portal-impl/src/content/Language_de.properties, ./portal-impl/src/content/Language_hu.properties]
        	at com.liferay.portal.kernel.language.LanguagePropertyTest._testValidKey(LanguagePropertyTest.java:286)
        	at com.liferay.portal.kernel.language.LanguagePropertyTest.testValidKeyUserDefaultPortrait(LanguagePropertyTest.java:168)
        
      2. LanguagePropertyTest.testUserNameRequiredFieldNamesSubsetOfUserNameFieldNames
        junit.framework.AssertionFailedError: Required field names are not a subset of user name field names in [./portal-impl/src/content/Language_nl_BE.properties, ./portal-impl/src/content/Language_es.properties, ./portal-impl/src/content/Language_zh_CN.properties, ./portal-impl/src/content/Language_fr.properties, ./portal-impl/src/content/Language_ca.properties, ./portal-impl/src/content/Language_ja.properties, ./portal-impl/src/content/Language_nl.properties, ./portal-impl/src/content/Language_pt_PT.properties, ./portal-impl/src/content/Language_de.properties, ./portal-impl/src/content/Language_hu.properties, ./portal-impl/src/content/Language_it.properties]
        	at com.liferay.portal.kernel.language.LanguagePropertyTest.testUserNameRequiredFieldNamesSubsetOfUserNameFieldNames(LanguagePropertyTest.java:145)
        
      3. LanguagePropertyTest.testValidKeyUserInitialsFieldNames
        junit.framework.AssertionFailedError: Invalid values for key "lang.user.initials.field.names" are found in: [./portal-impl/src/content/Language_nl_BE.properties, ./portal-impl/src/content/Language_es.properties, ./portal-impl/src/content/Language_fr.properties, ./portal-impl/src/content/Language_ca.properties, ./portal-impl/src/content/Language_nl.properties, ./portal-impl/src/content/Language_pt_PT.properties, ./portal-impl/src/content/Language_de.properties]
        	at com.liferay.portal.kernel.language.LanguagePropertyTest._testValidKey(LanguagePropertyTest.java:286)
        	at com.liferay.portal.kernel.language.LanguagePropertyTest.testValidKeyUserInitialsFieldNames(LanguagePropertyTest.java:173)
        
      4. ...

@shuyangzhou
Copy link
Owner

See #9544

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants