Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8255493: Support for pre-generated java.lang.invoke classes in CDS dynamic archive #3611

Closed
wants to merge 21 commits into from

Conversation

yminqi
Copy link
Contributor

@yminqi yminqi commented Apr 21, 2021

Hi, Please review

When do dynamic dump, the pre-generated lambda form classes from java.lang.invoke are not stored in dynamic archive. The patch will regenerate the four holder classes,
"java.lang.invoke.Invokers$Holder",
"java.lang.invoke.DirectMethodHandle$Holder",
"java.lang.invoke.DelegatingMethodHandle$Holder",
"java.lang.invoke.LambdaForm$Holder"
which will include the versions in static archive and new loaded functions all together and stored in dynamic archive. New test case added.
(Minor change to PrintSharedArchiveAtExit, which the index is not consecutive)

Tests: tier1,tier2,tier3,tier4

Thanks
Yumin


Progress

  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue
  • Change must be properly reviewed

Issue

  • JDK-8255493: Support for pre-generated java.lang.invoke classes in CDS dynamic archive

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/3611/head:pull/3611
$ git checkout pull/3611

Update a local copy of the PR:
$ git checkout pull/3611
$ git pull https://git.openjdk.java.net/jdk pull/3611/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 3611

View PR using the GUI difftool:
$ git pr show -t 3611

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/3611.diff

@bridgekeeper
Copy link

bridgekeeper bot commented Apr 21, 2021

👋 Welcome back minqi! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk openjdk bot added the rfr label Apr 21, 2021
@openjdk
Copy link

openjdk bot commented Apr 21, 2021

@yminqi The following label will be automatically applied to this pull request:

  • hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the hotspot label Apr 21, 2021
@mlbridge
Copy link

mlbridge bot commented Apr 21, 2021

@mlbridge
Copy link

mlbridge bot commented Apr 22, 2021

Mailing list message from David Holmes on hotspot-dev:

Hi Yumin,

On 22/04/2021 8:42 am, Yumin Qi wrote:

Hi, Please review

When do dynamic dump, the pre-generated lambda form classes from java.lang.invoke are not stored in dynamic archive. The patch will regenerate the four holder classes,
"java.lang.invoke.Invokers$Holder",
"java.lang.invoke.DirectMethodHandle$Holder",
"java.lang.invoke.DelegatingMethodHandle$Holder",
"java.lang.invoke.LambdaForm$Holder"
which will include the versions in static archive and new loaded functions all together and stored in dynamic archive. New test case added.
(Minor change to PrintSharedArchiveAtExit, which the index is not consecutive)

Meta questions:
- why are they not already stored?
- why do we need them to be stored?
- can we avoid hardwiring a set of classes that then tightly couples
our code to the JDK code in a way that we may not easily detect breakages?

Thanks,
David

1 similar comment
@mlbridge
Copy link

mlbridge bot commented Apr 22, 2021

Mailing list message from David Holmes on hotspot-dev:

Hi Yumin,

On 22/04/2021 8:42 am, Yumin Qi wrote:

Hi, Please review

When do dynamic dump, the pre-generated lambda form classes from java.lang.invoke are not stored in dynamic archive. The patch will regenerate the four holder classes,
"java.lang.invoke.Invokers$Holder",
"java.lang.invoke.DirectMethodHandle$Holder",
"java.lang.invoke.DelegatingMethodHandle$Holder",
"java.lang.invoke.LambdaForm$Holder"
which will include the versions in static archive and new loaded functions all together and stored in dynamic archive. New test case added.
(Minor change to PrintSharedArchiveAtExit, which the index is not consecutive)

Meta questions:
- why are they not already stored?
- why do we need them to be stored?
- can we avoid hardwiring a set of classes that then tightly couples
our code to the JDK code in a way that we may not easily detect breakages?

Thanks,
David

@yminqi
Copy link
Contributor Author

yminqi commented Apr 22, 2021

Mailing list message from David Holmes on hotspot-dev:

Hi Yumin,

On 22/04/2021 8:42 am, Yumin Qi wrote:

Hi, Please review
When do dynamic dump, the pre-generated lambda form classes from java.lang.invoke are not stored in dynamic archive. The patch will regenerate the four holder classes,
"java.lang.invoke.Invokers$Holder",
"java.lang.invoke.DirectMethodHandle$Holder",
"java.lang.invoke.DelegatingMethodHandle$Holder",
"java.lang.invoke.LambdaForm$Holder"
which will include the versions in static archive and new loaded functions all together and stored in dynamic archive. New test case added.
(Minor change to PrintSharedArchiveAtExit, which the index is not consecutive)

Meta questions:

  • why are they not already stored?
  • why do we need them to be stored?
  • can we avoid hardwiring a set of classes that then tightly couples
    our code to the JDK code in a way that we may not easily detect breakages?

Thanks,
David
Hi, David

  1. When dump static archive, we stored the holder classes which were logged (bug https://bugs.openjdk.java.net/browse/JDK-8247536) during build time. The pre-generated lambda form classes were collected in jlink stage, then regenerated the holder classes which hold the LF functions, stored in static archive. That is, whatever LF forms generated during build are archive in static archive.
  2. At runtime, user code Lambda will generate new classes which are not stored into dynamic archive. The holder classes are the four invoke class$holder classes, so we record the lambda format for dynamic dump, regenerate the holder classes which will host both static contents plus the newly loaded contents then stored in dynamic archive. There are two versions of those four holder classes, one is the static version, and the other (bigger) in dynamic, so if dynamic mapped, we use the dynamic versions which host more LF functions.
  3. We only regenerate the four holder classes and regardless of the others (species), those four holders host majority of the invoke functions. The names are hardcoded since there are no mapping from jdk to vm for this. Do we have a way to do this?

Thanks
Yumin

@mlbridge
Copy link

mlbridge bot commented Apr 22, 2021

Mailing list message from David Holmes on hotspot-dev:

On 22/04/2021 1:51 pm, Yumin Qi wrote:

_Mailing list message from [David Holmes](mailto:david.holmes at oracle.com) on [hotspot-dev](mailto:hotspot-dev at mail.openjdk.java.net):_

Hi Yumin,

On 22/04/2021 8:42 am, Yumin Qi wrote:

Hi, Please review
When do dynamic dump, the pre-generated lambda form classes from java.lang.invoke are not stored in dynamic archive. The patch will regenerate the four holder classes,
"java.lang.invoke.Invokers$Holder",
"java.lang.invoke.DirectMethodHandle$Holder",
"java.lang.invoke.DelegatingMethodHandle$Holder",
"java.lang.invoke.LambdaForm$Holder"
which will include the versions in static archive and new loaded functions all together and stored in dynamic archive. New test case added.
(Minor change to PrintSharedArchiveAtExit, which the index is not consecutive)

Meta questions:
- why are they not already stored?
- why do we need them to be stored?
- can we avoid hardwiring a set of classes that then tightly couples
our code to the JDK code in a way that we may not easily detect breakages?

Thanks,
David
Hi, David
1) When dump static archive, we stored the holder classes which were logged (bug https://bugs.openjdk.java.net/browse/JDK-8247536) during build time. The pre-generated lambda form classes were collected in jlink stage, then regenerated the holder classes which hold the LF functions, stored in static archive. That is, whatever LF forms generated during build are archive in static archive.

Okay I think I get how these classes are generated and then stored in
the static archive.

2) At runtime, user code Lambda will generate new classes which are not stored into dynamic archive. The holder classes are the four invoke class$holder classes, so we record the lambda format for dynamic dump, regenerate the holder classes which will host both static contents plus the newly loaded contents then stored in dynamic archive. There are two versions of those four holder classes, one is the static version, and the other (bigger) in dynamic, so if dynamic mapped, we use the dynamic versions which host more LF functions.

Not following this part. How can you regenerate an existing class? I'm
not familiar with the details of LF classes and how all that machinery
works.

3) We only regenerate the four holder classes and regardless of the others (species), those four holders host majority of the invoke functions. The names are hardcoded since there are no mapping from jdk to vm for this. Do we have a way to do this?

Probably not but that is the problem I'm flagging. Is this list of
classes going to grow? If they rework code on the Java side are we going
to know what changes to make on the VM side? Is way we can make this
less fragile?

Thanks,
David

1 similar comment
@mlbridge
Copy link

mlbridge bot commented Apr 22, 2021

Mailing list message from David Holmes on hotspot-dev:

On 22/04/2021 1:51 pm, Yumin Qi wrote:

_Mailing list message from [David Holmes](mailto:david.holmes at oracle.com) on [hotspot-dev](mailto:hotspot-dev at mail.openjdk.java.net):_

Hi Yumin,

On 22/04/2021 8:42 am, Yumin Qi wrote:

Hi, Please review
When do dynamic dump, the pre-generated lambda form classes from java.lang.invoke are not stored in dynamic archive. The patch will regenerate the four holder classes,
"java.lang.invoke.Invokers$Holder",
"java.lang.invoke.DirectMethodHandle$Holder",
"java.lang.invoke.DelegatingMethodHandle$Holder",
"java.lang.invoke.LambdaForm$Holder"
which will include the versions in static archive and new loaded functions all together and stored in dynamic archive. New test case added.
(Minor change to PrintSharedArchiveAtExit, which the index is not consecutive)

Meta questions:
- why are they not already stored?
- why do we need them to be stored?
- can we avoid hardwiring a set of classes that then tightly couples
our code to the JDK code in a way that we may not easily detect breakages?

Thanks,
David
Hi, David
1) When dump static archive, we stored the holder classes which were logged (bug https://bugs.openjdk.java.net/browse/JDK-8247536) during build time. The pre-generated lambda form classes were collected in jlink stage, then regenerated the holder classes which hold the LF functions, stored in static archive. That is, whatever LF forms generated during build are archive in static archive.

Okay I think I get how these classes are generated and then stored in
the static archive.

2) At runtime, user code Lambda will generate new classes which are not stored into dynamic archive. The holder classes are the four invoke class$holder classes, so we record the lambda format for dynamic dump, regenerate the holder classes which will host both static contents plus the newly loaded contents then stored in dynamic archive. There are two versions of those four holder classes, one is the static version, and the other (bigger) in dynamic, so if dynamic mapped, we use the dynamic versions which host more LF functions.

Not following this part. How can you regenerate an existing class? I'm
not familiar with the details of LF classes and how all that machinery
works.

3) We only regenerate the four holder classes and regardless of the others (species), those four holders host majority of the invoke functions. The names are hardcoded since there are no mapping from jdk to vm for this. Do we have a way to do this?

Probably not but that is the problem I'm flagging. Is this list of
classes going to grow? If they rework code on the Java side are we going
to know what changes to make on the VM side? Is way we can make this
less fragile?

Thanks,
David

@mlbridge
Copy link

mlbridge bot commented Apr 22, 2021

Mailing list message from David Holmes on hotspot-dev:

On 23/04/2021 7:18 am, Yumin Qi wrote:

On Thu, 22 Apr 2021 06:03:35 GMT, David Holmes <dholmes at openjdk.org> wrote:

src/hotspot/share/memory/dynamicArchive.cpp line 376:

374: assert(THREAD->is_Java_thread(), "Should be JavaThread");
375: log_info(cds, dynamic)("Regenerate lambdaform holder classes ...");
376: LambdaFormInvokers::regenerate_holder_classes(THREAD);

What if this throws an exception?

If there is exception in java side to regenerate the holder classes, there is a log for cds to indicate the regeneration failed, the exception is cleared and continue. It will not affect other parts.

There are two allocation sites in regenerate_holder_classes that use
CHECK and so will return if an OOME is pending. In those cases the
exception is neither logged nor cleared.

Cheers,
David
-----

1 similar comment
@mlbridge
Copy link

mlbridge bot commented Apr 22, 2021

Mailing list message from David Holmes on hotspot-dev:

On 23/04/2021 7:18 am, Yumin Qi wrote:

On Thu, 22 Apr 2021 06:03:35 GMT, David Holmes <dholmes at openjdk.org> wrote:

src/hotspot/share/memory/dynamicArchive.cpp line 376:

374: assert(THREAD->is_Java_thread(), "Should be JavaThread");
375: log_info(cds, dynamic)("Regenerate lambdaform holder classes ...");
376: LambdaFormInvokers::regenerate_holder_classes(THREAD);

What if this throws an exception?

If there is exception in java side to regenerate the holder classes, there is a log for cds to indicate the regeneration failed, the exception is cleared and continue. It will not affect other parts.

There are two allocation sites in regenerate_holder_classes that use
CHECK and so will return if an OOME is pending. In those cases the
exception is neither logged nor cleared.

Cheers,
David
-----

int len = _lambdaform_lines->length();
objArrayHandle list_lines = oopFactory::new_objArray_handle(vmClasses::String_klass(), len, CHECK);
if (list_lines == nullptr) {
Copy link
Member

@iklam iklam Apr 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no need to check for null. If allocation fails, the CHECK in the previous line will force this function to return.
(same for the check on line 96).

src/hotspot/share/cds/lambdaFormInvokers.cpp Outdated Show resolved Hide resolved
src/hotspot/share/cds/metaspaceShared.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@coleenp coleenp left a comment

I just had a couple of comments, otherwise looks ok to me but I have to confess that I don't know this code that well.

src/hotspot/share/cds/lambdaFormInvokers.cpp Outdated Show resolved Hide resolved
src/hotspot/share/cds/lambdaFormInvokers.cpp Outdated Show resolved Hide resolved
Copy link
Member

@calvinccheung calvinccheung left a comment

Looks good. Just a few minor comments.

@yminqi
Copy link
Contributor Author

yminqi commented Apr 27, 2021

Please hold, there is more clean up need to do: including file headers, empty lines etc.

_lambdaform_lines->append(line);
}

void LambdaFormInvokers::regenerate_holder_classes() {
Copy link
Member

@dholmes-ora dholmes-ora Apr 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is no longer a TRAPS function then it should not be using CHECK macro:

objArrayHandle list_lines = oopFactory::new_objArray_handle(vmClasses::String_klass(), len, CHECK);
  for (int i = 0; i < len; i++) {
Handle h_line = java_lang_String::create_from_str(_lambdaform_lines->at(i), CHECK);

If exceptions are not possible just use THREAD and comment why they are not possible. If they are possible then you need to use THREAD and handle them appropriately.

Copy link
Contributor Author

@yminqi yminqi Apr 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will rework the dynamic dump so it will be like static dump --- handle exceptions after returned. So will remove the all the checks in this file. Thanks.

@yminqi
Copy link
Contributor Author

yminqi commented Apr 29, 2021

The recent version change made DynamicArchive::dump take TRAPS, same as static dump. The design of LambdaFormInvokers::regenerate_holder_classes is that if the regeneration failed, it won't affect normal classes archived. So we have tests for wrong tag or line format in classlist to make sure dump archive. I changed to handle all exceptions except for OOM (which will be processed by caller if possible for the extreme case) in the function.

CLEAR_PENDING_EXCEPTION;
return;
}
HANDLE_IF_HAS_EXCEPTION;
Copy link
Member

@iklam iklam Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using a macro, I think the code can be more readable like this:

if (check_exception(THREAD)) {
    return;
}

and check_exception can contain the logic of HANDLE_IF_HAS_EXCEPTION.

Copy link
Contributor Author

@yminqi yminqi Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I was think of that, if a function or macro and finally took macro.

void LambdaFormInvokers::regenerate_holder_classes() {
#define HANDLE_IF_HAS_EXCEPTION \
if (HAS_PENDING_EXCEPTION) { \
if (!PENDING_EXCEPTION->is_a(vmClasses::OutOfMemoryError_klass())) { \
Copy link
Member

@iklam iklam Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest adding a comment and error message to explain why we ignore the error:

// We may get an exception if the classlist contains an error (or an outdated entry generated by
// an older JDK).
if (DumpSharedArchive) {
  log_error(cds)("Failed to generate LambdaForm holder classes. Is your classlist out of date?");
} else {
  log_error(cds)("Failed to generate LambdaForm holder classes. Was the base archive generated with an outdated classlist?");
}

Copy link
Contributor Author

@yminqi yminqi Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean DumpSharedSpaces? So the message will be different between static (DumpSharedSpaces) and dynamic, right?

Copy link
Member

@iklam iklam Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because the errors can only be caused by bad input from the classlist, which is only used during static archive dump. The information collected during dynamic dump are correct, and cannot cause CDS.generateLambdaFormHolderClasses() to fail.

CLEAR_PENDING_EXCEPTION;
return;
}
HANDLE_IF_HAS_EXCEPTION;
Copy link
Member

@iklam iklam Apr 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this necessary? I think bad classlists will cause errors in the JavaCalls::call_static call above, but not here.

Copy link
Contributor Author

@yminqi yminqi Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the original code, and it is necessary, KlassFactory::create_from_stream could throw exception.

Copy link
Member

@iklam iklam Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What kid of exceptions will be thrown (other than OOM?) Do you mean CDS.generateLambdaFormHolderClasses() can generate bad class files?

Copy link
Contributor Author

@yminqi yminqi Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is after CDS.generateLambdaFormHolderClasses, we get the byte stream of a class --- we need create instanceklass of the class. The exception is from ClassParser ---- whatever the exception during the process.

Copy link
Member

@iklam iklam Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ClassParser will throw an error only if OOM, or if the class is invalid. However, if the class is invalid, it means there's a bug inside generateLambdaFormHolderClasses() -- it's not supposed to generate an invalid class.

So we cannot just print a warning and ignore the error. The user may not notice the error message.

The archive dumping should fail so the user knows what has happened, and will hopefully report the error to us.

@@ -138,7 +140,6 @@ void LambdaFormInvokers::regenerate_holder_classes(TRAPS) {
memcpy(buf, (char*)h_bytes->byte_at_addr(0), len);
ClassFileStream st((u1*)buf, len, NULL, ClassFileStream::verify);
reload_class(class_name, st, THREAD);
Copy link
Member

@iklam iklam May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reload_class(class_name, st, THREAD); should be reload_class(class_name, st, CHECK);. If an error happens inside there, it means CDS.generateLambdaFormHolderClasses() has generated a bad classfile, or we have hit an OOM. In either case, we should report the error back to the caller.

Copy link
Contributor Author

@yminqi yminqi May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it should be a check here.

iklam
iklam approved these changes May 3, 2021
Copy link
Member

@iklam iklam left a comment

LGTM. Just a small nit about comments.

src/hotspot/share/cds/lambdaFormInvokers.hpp Outdated Show resolved Hide resolved
@openjdk
Copy link

openjdk bot commented May 3, 2021

@yminqi This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8255493: Support for pre-generated java.lang.invoke classes in CDS dynamic archive

Reviewed-by: iklam, ccheung

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 44 new commits pushed to the master branch:

  • 770dfc1: 8265279: Remove unused RandomGeneratorFactory.all(Class category)
  • ee5bba0: 8265767: compiler/eliminateAutobox/TestIntBoxing.java crashes on arm32 after 8264649 in debug VMs
  • 05e6017: 8265137: java.util.Random suddenly has new public methods nowhere documented
  • aa90df6: 8266187: Memory leak in appendBootClassPath()
  • b651904: 8266438: Compile::remove_useless_nodes does not remove opaque nodes
  • 141cc2f: 8261527: Record page size used for underlying mapping in ReservedSpace
  • 8e071c4: 8265784: [C2] Hoisting of DecodeN leaves MachTemp inputs behind
  • ce1bc9d: 8266432: ZGC: GC allocation stalls can trigger deadlocks
  • 30ccd80: 8264950: Set opaque for JTooltip in config file of NimbusLookAndFeel
  • cfdf4a7: 8266449: cleanup jtreg tags in compiler/intrinsics/sha/cli tests
  • ... and 34 more: https://git.openjdk.java.net/jdk/compare/51b218842f001f1c4fd5ca7a02a2ba21e9e8a82c...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready label May 3, 2021
Copy link
Member

@dholmes-ora dholmes-ora left a comment

The TRAPS/CHECK usages seem okay now.

Thanks,
David

@yminqi
Copy link
Contributor Author

yminqi commented May 4, 2021

@dholmes-ora Thanks for comment again. @calvinccheung @coleenp I need one more OK to push since this is not a trivial fix.

Copy link
Member

@calvinccheung calvinccheung left a comment

Latest version looks good. Just one nit.

Thanks,
Calvin

@yminqi
Copy link
Contributor Author

yminqi commented May 4, 2021

/integrate

@openjdk openjdk bot closed this May 4, 2021
@openjdk openjdk bot added integrated and removed ready rfr labels May 4, 2021
@openjdk
Copy link

openjdk bot commented May 4, 2021

@yminqi Since your change was applied there have been 44 commits pushed to the master branch:

  • 770dfc1: 8265279: Remove unused RandomGeneratorFactory.all(Class category)
  • ee5bba0: 8265767: compiler/eliminateAutobox/TestIntBoxing.java crashes on arm32 after 8264649 in debug VMs
  • 05e6017: 8265137: java.util.Random suddenly has new public methods nowhere documented
  • aa90df6: 8266187: Memory leak in appendBootClassPath()
  • b651904: 8266438: Compile::remove_useless_nodes does not remove opaque nodes
  • 141cc2f: 8261527: Record page size used for underlying mapping in ReservedSpace
  • 8e071c4: 8265784: [C2] Hoisting of DecodeN leaves MachTemp inputs behind
  • ce1bc9d: 8266432: ZGC: GC allocation stalls can trigger deadlocks
  • 30ccd80: 8264950: Set opaque for JTooltip in config file of NimbusLookAndFeel
  • cfdf4a7: 8266449: cleanup jtreg tags in compiler/intrinsics/sha/cli tests
  • ... and 34 more: https://git.openjdk.java.net/jdk/compare/51b218842f001f1c4fd5ca7a02a2ba21e9e8a82c...master

Your commit was automatically rebased without conflicts.

Pushed as commit 8b37d48.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hotspot integrated
5 participants